Oct 30 13:48:14.336814 ip-10-0-133-118 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 13:48:14.707010 ip-10-0-133-118 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:48:14.707010 ip-10-0-133-118 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 13:48:14.707010 ip-10-0-133-118 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:48:14.707010 ip-10-0-133-118 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:48:14.709984 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.709915    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:14.709984 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.709984    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:14.709984 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.709987    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.709990    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.709993    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.709997    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710000    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710003    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710005    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710008    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710010    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710013    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710016    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710024    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710027    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710029    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710032    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710035    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710037    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710040    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710043    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:14.710072 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710045    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710048    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710051    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710053    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710056    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710058    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710061    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710063    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710066    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710068    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710071    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710073    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710076    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710079    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710082    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710085    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710088    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710090    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710093    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:14.710496 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710095    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710098    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710100    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710103    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710105    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710108    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710110    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710113    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710116    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710118    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710123    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710126    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710129    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710131    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710134    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710139    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710144    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710147    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710150    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710153    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:14.711115 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710155    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710158    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710161    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710164    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710166    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710169    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710171    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710176    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710179    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710182    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710185    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710188    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710191    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710193    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710196    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710198    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710201    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710204    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710206    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710210    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:14.711752 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710212    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710215    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710217    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710220    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710223    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710226    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710228    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710233    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710235    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710239    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710241    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710244    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710247    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710249    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710252    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710254    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710257    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710259    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710263    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710266    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:14.712254 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710268    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:14.712746 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710271    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:14.712746 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710273    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:14.712746 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710276    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:14.712746 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710278    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:14.712746 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.710281    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:14.713670 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713648    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:14.713670 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713666    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:14.713670 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713671    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:14.713670 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713675    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713680    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713684    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713688    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713692    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713696    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713699    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713704    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713708    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713712    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713716    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713720    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713724    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713728    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713732    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713736    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713740    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713743    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713747    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713753    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:14.713905 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713760    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713765    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713769    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713773    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713778    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713781    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713785    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713790    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713794    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713799    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713803    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713807    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713811    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713815    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713820    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713825    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713829    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713834    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713839    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:14.714820 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713843    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713848    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713852    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713856    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713860    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713865    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713869    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713873    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713877    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713882    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713886    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713890    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713894    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713898    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713902    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713907    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713911    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713915    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713919    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713924    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:14.715664 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713928    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713933    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713937    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713941    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713945    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713950    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713955    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713960    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713965    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713969    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713973    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713977    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713982    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713986    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713991    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.713995    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714000    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714006    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714010    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714015    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:14.716158 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714019    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714032    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714037    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714041    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714045    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714050    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714054    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714059    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714063    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714067    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714072    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714076    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714080    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714085    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714089    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714096    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714101    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714107    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714112    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:14.716861 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714119    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714123    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714127    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714132    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.714136    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715024    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715033    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715038    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715042    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715047    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715051    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715055    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715060    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715064    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715077    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715081    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715085    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715090    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715094    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:14.717713 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715098    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715103    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715107    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715112    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715116    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715120    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715125    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715129    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715134    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715138    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715142    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715146    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715150    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715157    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715165    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715171    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715176    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715180    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715184    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715189    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:14.718527 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715193    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715197    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715202    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715206    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715210    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715214    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715218    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715222    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715235    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715239    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715244    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715248    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715252    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715256    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715262    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715266    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715271    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715275    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715279    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715283    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:14.719203 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715287    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715292    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715296    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715300    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715304    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715308    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715313    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715318    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715322    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715326    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715331    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715335    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715339    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715343    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715347    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715351    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715355    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715359    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715364    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:14.719835 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715368    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715372    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715384    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715388    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715392    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715397    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715401    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715405    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715409    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715413    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715417    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715421    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715426    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715430    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715434    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715441    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715446    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715450    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715458    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:14.720340 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715462    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715467    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715471    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715475    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715479    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715484    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715488    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715492    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715497    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715501    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715510    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715515    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715519    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.715523    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716459    2580 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716473    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716498    2580 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716506    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716512    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716517    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716524    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 13:48:14.720909 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716531    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716536    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716560    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716565    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716572    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716578    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716582    2580 flags.go:64] FLAG: --cgroup-root=""
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716587    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716592    2580 flags.go:64] FLAG: --client-ca-file=""
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716596    2580 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716601    2580 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716644    2580 flags.go:64] FLAG: --cluster-domain=""
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716649    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716654    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716660    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716666    2580 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716672    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716677    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716682    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716687    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716692    2580 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716696    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716701    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716707    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 13:48:14.721423 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716712    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716718    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716723    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716728    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716740    2580 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716746    2580 flags.go:64] FLAG: --enable-server="true"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716750    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716763    2580 flags.go:64] FLAG: --event-burst="100"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716768    2580 flags.go:64] FLAG: --event-qps="50"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716773    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716778    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716783    2580 flags.go:64] FLAG: --eviction-hard=""
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716790    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716795    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716800    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716805    2580 flags.go:64] FLAG: --eviction-soft=""
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716810    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716815    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716819    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716824    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716829    2580 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716834    2580 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716839    2580 flags.go:64] FLAG: --feature-gates=""
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716846    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716851    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 13:48:14.722155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716856    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716861    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716865    2580 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716870    2580 flags.go:64] FLAG: --help="false"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716875    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-133-118.ec2.internal"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716879    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716884    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716889    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716894    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716901    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716905    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716910    2580 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716923    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716941    2580 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716951    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716957    2580 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716962    2580 flags.go:64] FLAG: --kube-reserved=""
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716967    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716971    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716976    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716981    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716985    2580 flags.go:64] FLAG: --lock-file=""
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716990    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.716995    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 13:48:14.722778 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717000    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717009    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717014    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717019    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717024    2580 flags.go:64] FLAG: --logging-format="text"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717029    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717034    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717039    2580 flags.go:64] FLAG: --manifest-url=""
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717043    2580 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717050    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717055    2580 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717062    2580 flags.go:64] FLAG: --max-pods="110"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717067    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717072    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717076    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717081    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717086    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717091    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717095    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717106    2580 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717111    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717124    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717131    2580 flags.go:64] FLAG: --pod-cidr=""
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717137    2580 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717141    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 13:48:14.723343 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717147    2580 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717151    2580 flags.go:64] FLAG: --port="10250"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717156    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717161    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ec13e27d2b23feb0"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717166    2580 flags.go:64] FLAG: --qos-reserved=""
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717171    2580 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717176    2580 flags.go:64] FLAG: --register-node="true"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717180    2580 flags.go:64] FLAG: --register-with-taints=""
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717193    2580 flags.go:64] FLAG: --registry-burst="10"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717198    2580 flags.go:64] FLAG: --registry-qps="5"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717202    2580 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717207    2580 flags.go:64] FLAG: --reserved-memory=""
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717213    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717218    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717223    2580 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717228    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717232    2580 flags.go:64] FLAG: --runonce="false"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717238    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717243    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717248    2580 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717253    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717257    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717262    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717267    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717272    2580 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717277    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 13:48:14.724001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717282    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717286    2580 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717291    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717305    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717310    2580 flags.go:64] FLAG: --system-cgroups=""
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717315    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717324    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717328    2580 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717333    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717343    2580 flags.go:64] FLAG: --tls-min-version=""
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717348    2580 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717352    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717357    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717362    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717369    2580 flags.go:64] FLAG: --v="2"
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717376    2580 flags.go:64] FLAG: --version="false"
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717382    2580 flags.go:64] FLAG: --vmodule=""
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717388    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.717394    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717684    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717693    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717698    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717702    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717707    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:14.724640 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717711    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717716    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717720    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717724    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717728    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717732    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717736    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717741    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717745    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717749    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717754    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717760    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717775    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717779    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717783    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717789    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717793    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717797    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717802    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717806    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:14.725198 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717810    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717814    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717821    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717825    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717830    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717834    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717838    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717843    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717847    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717851    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717855    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717859    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717863    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717868    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717872    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717876    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717881    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717885    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717890    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717894    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:14.726242 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717899    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717903    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717907    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717913    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717918    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717930    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717934    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717938    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717942    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717947    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717952    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717957    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717961    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717965    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717971    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717975    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717980    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717984    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717988    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717992    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:14.726744 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.717997    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718002    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718006    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718011    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718015    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718020    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718024    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718028    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718033    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718037    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718041    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718045    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718050    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718054    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718058    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718064    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718068    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718073    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718085    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:14.727208 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718090    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718094    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718101    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718106    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718111    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718115    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718120    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718126    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718130    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718134    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718139    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718143    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718148    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718152    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718156    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718161    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718165    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718171    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718178    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:14.727772 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718184    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718188    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718192    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.718236    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718304    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718311    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718316    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718321    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718325    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718330    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718336    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718340    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718345    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718350    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718354    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718359    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:14.728220 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718363    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718368    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718372    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718376    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718383    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718387    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718391    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718395    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718400    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718404    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718408    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718413    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718417    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718421    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718427    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718433    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718438    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718442    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718446    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:14.728618 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718451    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718455    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718460    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718464    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718468    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718473    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718477    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718484    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718488    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718493    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718500    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718506    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718510    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718515    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718519    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718523    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718527    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718533    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718551    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:14.729059 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718556    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718560    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718564    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718568    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718572    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718577    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718581    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718585    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718590    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718594    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718598    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718602    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718606    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718611    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718615    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718620    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718625    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718630    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718634    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718638    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:14.729494 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718643    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718647    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718651    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718655    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718659    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718664    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718669    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718673    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718677    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718681    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718687    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718691    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718695    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718700    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718704    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718708    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718713    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718717    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718721    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718725    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:14.729989 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718730    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718734    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718739    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718743    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718748    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718752    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718756    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718761    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718765    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718769    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718773    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718781    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718786    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718790    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718794    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.718798    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.725244    2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.725259    2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.725807    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.725814    2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 13:48:14.730998 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725867    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725872    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725875    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725878    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725881    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725883    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725886    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725889    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725891    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725895    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725897    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725900    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725902    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725905    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725908    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725911    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725913    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725916    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725919    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:14.731471 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725921    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725924    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725927    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725930    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725933    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725936    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725938    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725941    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725943    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725946    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725955    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725958    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725961    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725964    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725966    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725969    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725972    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725974    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725977    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725979    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:14.731966 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725982    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725984    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725987    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725989    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725993    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725995    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.725998    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726000    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726003    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726005    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726008    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726011    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726014    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726016    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726019    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726022    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726025    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726028    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726030    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:14.732432 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726033    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726036    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726038    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726041    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726049    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726052    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726055    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726058    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726060    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726063    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726065    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726068    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726070    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726073    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726076    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726078    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726081    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726083    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726086    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726089    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:14.733164 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726092    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726095    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726098    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726100    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726103    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726105    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726108    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726111    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726114    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726116    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726119    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726121    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726123    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726126    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726128    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726132    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726137    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726140    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726143    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:14.733663 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726146    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726149    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726153    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726155    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726158    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726160    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726163    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726165    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726168    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.726195    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726334    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726338    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726340    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726343    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726346    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:14.734421 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726348    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726351    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726354    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726356    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726359    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726361    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726364    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726366    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726369    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726371    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726374    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726376    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726379    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726381    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726384    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726387    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726390    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726392    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726395    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:14.734893 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726397    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726400    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726402    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726405    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726408    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726411    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726413    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726415    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726418    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726421    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726423    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726425    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726428    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726430    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726433    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726436    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726438    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726440    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726443    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726446    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:14.735395 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726448    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726451    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726453    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726456    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726459    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726461    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726464    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726466    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726469    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726471    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726474    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726477    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726481    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726484    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726487    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726490    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726493    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726497    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726500    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:14.736050 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726503    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726506    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726509    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726511    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726514    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726516    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726519    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726521    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726524    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726526    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726528    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726532    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726534    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726557    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726562    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726566    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726570    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726572    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726575    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726578    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:14.736579 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726580    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726583    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726585    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726588    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726591    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726594    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726596    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726599    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726601    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726604    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726606    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726609    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726611    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726614    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726617    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726620    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726622    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726625    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726627    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:14.737068 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726630    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726632    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726635    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726637    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.726664    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726701    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726705    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726708    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726711    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726713    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726716    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726718    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726721    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726724    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726726    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726729    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:14.737506 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726732    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726735    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726738    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726740    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726743    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726745    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726748    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726751    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726754    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726757    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726759    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726762    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726766    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726769    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726771    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726774    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726776    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726779    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726781    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726784    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:14.738055 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726788    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726790    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726793    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726796    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726798    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726801    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726803    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726806    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726809    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726811    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726814    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726816    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726819    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726822    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726824    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726828    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726832    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726836    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726839    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726842    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:14.738529 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726845    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726849    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726852    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726855    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726861    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726863    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726866    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726869    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726871    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726874    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726877    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726879    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726883    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726886    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726889    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726891    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726894    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726896    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726899    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:14.739146 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726901    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726904    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726907    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726910    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726913    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726916    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726918    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726921    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726923    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726926    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726929    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726932    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726934    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726937    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726940    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726942    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726945    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726948    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726951    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:14.739659 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726954    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726957    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726960    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726963    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726966    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726968    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726972    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726975    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726977    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726980    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726983    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726985    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726988    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726990    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726993    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726996    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:14.726999    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.727059    2580 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.728881    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 13:48:14.740135 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.729740    2580 server.go:1014] "Starting client certificate rotation"
Oct 30 13:48:14.740633 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.729831    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 13:48:14.740633 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.729864    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 13:48:14.740633 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.731487    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:48:14.740633 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.733004    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:48:14.742676 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.742659    2580 log.go:25] "Validated CRI v1 runtime API"
Oct 30 13:48:14.745581 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.745566    2580 log.go:25] "Validated CRI v1 image API"
Oct 30 13:48:14.746290 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.746262    2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 13:48:14.752007 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.751988    2580 fs.go:135] Filesystem UUIDs: map[1b383948-1a3b-4c69-85e9-a21c409e3e12:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f488d9b9-b66f-4f13-8cdf-ac9e22d4ec79:/dev/nvme0n1p3]
Oct 30 13:48:14.752071 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.752006    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 13:48:14.757685 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.757579    2580 manager.go:217] Machine: {Timestamp:2026-10-30 13:48:14.755601843 +0000 UTC m=+0.314604081 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107327 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d23b13ea07d5f567963215179f05c SystemUUID:ec2d23b1-3ea0-7d5f-5679-63215179f05c BootID:f3fb7500-0f47-446e-9dc2-5ff832c24c71 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e3:7a:80:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e3:7a:80:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:5f:80:91:a3:cc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 13:48:14.757685 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.757682    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 13:48:14.757794 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.757778    2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 13:48:14.758897 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.758871    2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 13:48:14.759024 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.758897    2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-118.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 13:48:14.759453 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.759443    2580 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 13:48:14.759491 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.759454    2580 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 13:48:14.759491 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.759477    2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 13:48:14.759491 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.759490    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 13:48:14.760206 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.760197    2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 13:48:14.763774 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.763762    2580 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 13:48:14.763882 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.763873    2580 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 13:48:14.764498 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.764489    2580 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 13:48:14.764570 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.764501    2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 13:48:14.764570 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.764510    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 13:48:14.764570 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.764521    2580 kubelet.go:405] "Adding apiserver pod source"
Oct 30 13:48:14.764570 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.764530    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 13:48:14.764835 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.764821    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 13:48:14.766855 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.766840    2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 13:48:14.771054 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.771039    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 13:48:14.771930 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.771916    2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 13:48:14.772001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.771956    2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 13:48:14.772153 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.772138    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 13:48:14.772216 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.772167    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 13:48:14.772216 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.772182    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 13:48:14.772216 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.772195    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 13:48:14.772216 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.772210    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 13:48:14.772324 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.772225    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 13:48:14.772953 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.772943    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 13:48:14.773001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.772957    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 13:48:14.773001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.772968    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 13:48:14.773001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.772979    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 13:48:14.773001 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.772993    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 13:48:14.773222 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.773008    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 13:48:14.773611 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.773599    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 13:48:14.773669 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.773614    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 13:48:14.777210 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.777194    2580 server.go:1263] "Started kubelet"
Oct 30 13:48:14.777426 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.777393    2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 13:48:14.777481 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.777363    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 13:48:14.777637 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.777622    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 13:48:14.777919 ip-10-0-133-118 systemd[1]: Started Kubernetes Kubelet.
Oct 30 13:48:14.778849 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.778759    2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 13:48:14.780426 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.780401    2580 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 13:48:14.784392 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:14.784373    2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 13:48:14.786407 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.786387    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 13:48:14.786912 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.786893    2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 13:48:14.787010 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:14.786974    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:14.787773 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.787754    2580 factory.go:55] Registering systemd factory
Oct 30 13:48:14.787773 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.787775    2580 factory.go:223] Registration of the systemd container factory successfully
Oct 30 13:48:14.787963 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.787945    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 13:48:14.787963 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.787956    2580 factory.go:153] Registering CRI-O factory
Oct 30 13:48:14.788091 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.787972    2580 factory.go:223] Registration of the crio container factory successfully
Oct 30 13:48:14.788091 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.788020    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 13:48:14.788091 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.788044    2580 factory.go:103] Registering Raw factory
Oct 30 13:48:14.788091 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.787947    2580 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 13:48:14.788091 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.788066    2580 manager.go:1199] Started watching for new ooms in manager
Oct 30 13:48:14.788091 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.788069    2580 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 13:48:14.788275 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.788146    2580 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 13:48:14.788275 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.788152    2580 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 13:48:14.788936 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.788922    2580 manager.go:319] Starting recovery of all containers
Oct 30 13:48:14.795879 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.795845    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 13:48:14.798653 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.798633    2580 manager.go:324] Recovery completed
Oct 30 13:48:14.804512 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.804498    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:48:14.806145 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.806118    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:48:14.806207 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.806164    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:48:14.806207 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.806179    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:48:14.806715 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.806700    2580 cpu_manager.go:226] "Starting" policy="none"
Oct 30 13:48:14.806715 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.806712    2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 13:48:14.806831 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.806731    2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 13:48:14.806831 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.806768    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:48:14.807534 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.807520    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:48:14.807607 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.807600    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:48:14.807661 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.807618    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:48:14.808810 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.808798    2580 policy_none.go:50] "Start"
Oct 30 13:48:14.808874 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.808813    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 13:48:14.808874 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.808838    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 13:48:14.810853 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.810843    2580 policy_none.go:44] "Start"
Oct 30 13:48:14.813687 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.813662    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-118.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 13:48:14.813771 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:14.813672    2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 13:48:14.814569 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:14.813740    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-118.ec2.internal.18e352c7d0b54888  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-118.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-118.ec2.internal,},FirstTimestamp:2026-10-30 13:48:14.777100424 +0000 UTC m=+0.336102663,LastTimestamp:2026-10-30 13:48:14.777100424 +0000 UTC m=+0.336102663,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-118.ec2.internal,}"
Oct 30 13:48:14.822940 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.822920    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 13:48:14.823853 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.823837    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8bcrd"
Oct 30 13:48:14.828834 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:14.828805    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 13:48:14.828984 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:14.828962    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 13:48:14.837003 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.836986    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8bcrd"
Oct 30 13:48:14.842165 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:14.842134    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 13:48:14.844093 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.844066    2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 13:48:14.847032 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.847020    2580 manager.go:341] "Starting Device Plugin manager"
Oct 30 13:48:14.847125 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:14.847048    2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 13:48:14.847125 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.847056    2580 server.go:87] "Starting device plugin registration server"
Oct 30 13:48:14.847252 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.847239    2580 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 13:48:14.847303 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.847252    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 13:48:14.847405 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.847394    2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 13:48:14.847477 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.847457    2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 13:48:14.847528 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.847483    2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 13:48:14.847898 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:14.847875    2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 13:48:14.847987 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:14.847924    2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:14.919175 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.919155    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 13:48:14.919304 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.919179    2580 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 13:48:14.919304 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.919199    2580 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 13:48:14.919304 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:14.919238    2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 13:48:14.935441 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.935419    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:48:14.947932 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.947915    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:48:14.951578 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.951562    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:48:14.951645 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.951588    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:48:14.951645 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.951598    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:48:14.951645 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.951621    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-118.ec2.internal"
Oct 30 13:48:14.969604 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:14.969530    2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-118.ec2.internal"
Oct 30 13:48:14.969604 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:14.969569    2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-118.ec2.internal\": node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:15.006997 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:15.006976    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:15.020216 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.020198    2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"]
Oct 30 13:48:15.020292 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.020263    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:48:15.021043 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.021024    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:48:15.021118 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.021050    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:48:15.021118 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.021062    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:48:15.023262 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.023250    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:48:15.023419 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.023406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.023482 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.023434    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:48:15.023957 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.023940    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:48:15.024039 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.023968    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:48:15.024039 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.023979    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:48:15.024039 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.023943    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:48:15.024039 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.024035    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:48:15.024187 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.024045    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:48:15.026010 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.025996    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.026071 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.026020    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:48:15.026665 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.026652    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:48:15.026733 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.026678    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:48:15.026733 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.026687    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:48:15.056055 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:15.056034    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-118.ec2.internal\" not found" node="ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.058588 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:15.058573    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-118.ec2.internal\" not found" node="ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.090886 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.090868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b30ac2208b235a46404847e29afd4a13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"b30ac2208b235a46404847e29afd4a13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.090940 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.090891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b30ac2208b235a46404847e29afd4a13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"b30ac2208b235a46404847e29afd4a13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.090940 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.090913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e329b8d309e1f8141c44f91d4bb26de8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-118.ec2.internal\" (UID: \"e329b8d309e1f8141c44f91d4bb26de8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.107265 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:15.107248    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:15.191040 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.191020    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b30ac2208b235a46404847e29afd4a13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"b30ac2208b235a46404847e29afd4a13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.191094 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.190984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b30ac2208b235a46404847e29afd4a13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"b30ac2208b235a46404847e29afd4a13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.191094 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.191083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e329b8d309e1f8141c44f91d4bb26de8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-118.ec2.internal\" (UID: \"e329b8d309e1f8141c44f91d4bb26de8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.191154 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.191098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b30ac2208b235a46404847e29afd4a13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"b30ac2208b235a46404847e29afd4a13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.191154 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.191133    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b30ac2208b235a46404847e29afd4a13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"b30ac2208b235a46404847e29afd4a13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.191214 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.191184    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e329b8d309e1f8141c44f91d4bb26de8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-118.ec2.internal\" (UID: \"e329b8d309e1f8141c44f91d4bb26de8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.207594 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:15.207574    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:15.307837 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:15.307821    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:15.359477 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.359459    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.361607 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.361594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:15.408377 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:15.408250    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:15.508817 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:15.508784    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:15.609340 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:15.609273    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:15.709808 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:15.709782    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:15.729958 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.729936    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 13:48:15.730077 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.730060    2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 13:48:15.810732 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:15.810710    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:15.842066 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.842037    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 13:43:15 +0000 UTC" deadline="2028-08-02 01:48:33.600702944 +0000 UTC"
Oct 30 13:48:15.842066 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.842061    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15396h0m17.75864478s"
Oct 30 13:48:15.849285 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.849057    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:48:15.850800 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.850366    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:48:15.850800 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.850403    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:48:15.850800 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.850420    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:48:15.911715 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:15.911662    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:15.920897 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.920866    2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:48:15.961151 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:15.961114    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode329b8d309e1f8141c44f91d4bb26de8.slice/crio-c71db214edb721aea8aa75d032c94def95d0604d9ae341a7dc3e163700c4777a WatchSource:0}: Error finding container c71db214edb721aea8aa75d032c94def95d0604d9ae341a7dc3e163700c4777a: Status 404 returned error can't find the container with id c71db214edb721aea8aa75d032c94def95d0604d9ae341a7dc3e163700c4777a
Oct 30 13:48:15.961609 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:15.961568    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30ac2208b235a46404847e29afd4a13.slice/crio-1fc3828e972ad5a88dd2594e0093f91696fc979c397cba6d0a48747b9a081914 WatchSource:0}: Error finding container 1fc3828e972ad5a88dd2594e0093f91696fc979c397cba6d0a48747b9a081914: Status 404 returned error can't find the container with id 1fc3828e972ad5a88dd2594e0093f91696fc979c397cba6d0a48747b9a081914
Oct 30 13:48:15.967331 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:15.967313    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:48:16.012097 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:16.012074    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:16.112978 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:16.112948    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:16.213561 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:16.213501    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:16.313911 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:16.313888    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:16.375066 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.375042    2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:48:16.388188 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.388163    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:16.404933 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.404911    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:48:16.416654 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.416635    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:48:16.417641 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.417614    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:16.443030 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.443005    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:48:16.765618 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.765588    2580 apiserver.go:52] "Watching apiserver"
Oct 30 13:48:16.787632 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.787605    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 13:48:16.808926 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.808901    2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 13:48:16.809239 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.809218    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-kgvgs","kube-system/global-pull-secret-syncer-2w5dd","openshift-multus/network-metrics-daemon-l5x6g","openshift-network-diagnostics/network-check-target-98cgg","kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qlh2v","openshift-image-registry/node-ca-6pbvr","openshift-network-operator/iptables-alerter-czxww","kube-system/konnectivity-agent-sctxm","openshift-dns/node-resolver-cjg7z","openshift-multus/multus-additional-cni-plugins-g5xcz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal","openshift-multus/multus-rjz78"]
Oct 30 13:48:16.812222 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.812150    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:16.814282 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.814260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:16.818806 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.818782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:16.818947 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:16.818891    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:16.821155 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.821126    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:16.821257 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:16.821202    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:16.823765 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.823741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.831025 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.831005    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 13:48:16.831148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.831131    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jc894\""
Oct 30 13:48:16.831270 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.831249    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 13:48:16.831370 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.831295    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 13:48:16.831370 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.831336    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.831455 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.831377    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:16.832978 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.832952    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.833575 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.833556    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2dbfk\""
Oct 30 13:48:16.835720 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.835697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:16.835811 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:16.835774    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:16.838041 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.838013    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:16.838900 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.838877    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 13:48:16.838996 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.838945    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hr7nb"
Oct 30 13:48:16.841088 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.841070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:16.843699 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.843678    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:16.852094 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.852075    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hr7nb"
Oct 30 13:48:16.889649 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.889610    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 13:48:16.905023 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.904994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-modprobe-d\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.905130 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.905065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-daemon-config\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.905130 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.905101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-etc-kubernetes\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.905253 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.905133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-env-overrides\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.905253 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.905181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-ovnkube-script-lib\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.905253 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.905214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-tmp-dir\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:16.907122 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.907094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2-agent-certs\") pod \"konnectivity-agent-sctxm\" (UID: \"efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2\") " pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:16.907218 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.907184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k6s7\" (UniqueName: \"kubernetes.io/projected/055c8a12-fcb0-440d-bacc-20dd67b2ce02-kube-api-access-6k6s7\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:16.907281 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.907260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz5fm\" (UniqueName: \"kubernetes.io/projected/9acb4199-6d58-4d3d-86dc-96326517baf8-kube-api-access-lz5fm\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:16.907338 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.907316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-cni-bin\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.907390 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.907376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-lib-modules\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.907552 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.907478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-slash\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.907634 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.907613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-systemd\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.907736 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.907712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-cni-bin\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.907797 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.907773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxql\" (UniqueName: \"kubernetes.io/projected/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-kube-api-access-xpxql\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:16.908047 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-tuned\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.908109 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-cnibin\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.908160 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908128    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwg4\" (UniqueName: \"kubernetes.io/projected/375bd1fd-307b-4a6b-b6be-99b492065982-kube-api-access-tdwg4\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:16.908391 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b3dcf54-ba8f-4080-9666-cb41a151ccad-tmp\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.908443 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2-konnectivity-ca\") pod \"konnectivity-agent-sctxm\" (UID: \"efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2\") " pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:16.908494 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-sys\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.908579 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-netns\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.908642 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.908706 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-ovn\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.908756 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-kubernetes\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.908756 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-socket-dir-parent\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.908857 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-conf-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.908857 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysctl-d\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.908955 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp28b\" (UniqueName: \"kubernetes.io/projected/0b3dcf54-ba8f-4080-9666-cb41a151ccad-kube-api-access-rp28b\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.908955 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-cni-binary-copy\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.909044 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.908962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-hostroot\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.909093 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.909093 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cni-binary-copy\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:16.909193 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-socket-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:16.909193 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:16.909290 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-cni-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.909341 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.909390 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvkx\" (UniqueName: \"kubernetes.io/projected/d7926b85-83e3-4e81-a403-774ea8daebe1-kube-api-access-8zvkx\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.909390 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055c8a12-fcb0-440d-bacc-20dd67b2ce02-host-slash\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:16.909497 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44fn2\" (UniqueName: \"kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2\") pod \"network-check-target-98cgg\" (UID: \"fafdd535-5a88-4207-b2a4-cc43bf43bd17\") " pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:16.909497 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rnlm\" (UniqueName: \"kubernetes.io/projected/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-kube-api-access-2rnlm\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.909497 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-log-socket\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.909647 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-cni-netd\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.909647 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0bbec59-52f9-4dc6-b24b-169f666c08ae-dbus\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:16.909647 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-hosts-file\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:16.909784 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/055c8a12-fcb0-440d-bacc-20dd67b2ce02-iptables-alerter-script\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:16.909832 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-k8s-cni-cncf-io\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.909884 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-kubelet\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.909940 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:16.909990 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:16.910039 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.909986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6hc\" (UniqueName: \"kubernetes.io/projected/08311d3a-d790-4761-96f0-e9a6b336e913-kube-api-access-9f6hc\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:16.910039 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-run\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.910127 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-var-lib-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.910127 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-node-log\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.910220 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7926b85-83e3-4e81-a403-774ea8daebe1-ovn-node-metrics-cert\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.910220 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:16.910220 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:16.910362 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-kubelet\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.910362 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cnibin\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:16.910454 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:16.910454 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-multus-certs\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.910547 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysconfig\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.910547 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-systemd\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.910670 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-cni-multus\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.910918 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-system-cni-dir\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:16.911029 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.910957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-etc-selinux\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:16.911457 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-sys-fs\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:16.911579 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-run-netns\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.911579 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-var-lib-kubelet\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.911579 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysctl-conf\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.911579 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-system-cni-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:16.911769 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-ovnkube-config\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.911769 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-host\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:16.911769 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-systemd-units\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.911769 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfk95\" (UniqueName: \"kubernetes.io/projected/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-kube-api-access-kfk95\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:16.911769 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-device-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:16.911769 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-etc-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:16.911983 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0bbec59-52f9-4dc6-b24b-169f666c08ae-kubelet-config\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:16.911983 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-registration-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:16.911983 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08311d3a-d790-4761-96f0-e9a6b336e913-host\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:16.911983 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.911854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08311d3a-d790-4761-96f0-e9a6b336e913-serviceca\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:16.924632 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.924572    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" event={"ID":"b30ac2208b235a46404847e29afd4a13","Type":"ContainerStarted","Data":"1fc3828e972ad5a88dd2594e0093f91696fc979c397cba6d0a48747b9a081914"}
Oct 30 13:48:16.925948 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.925921    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" event={"ID":"e329b8d309e1f8141c44f91d4bb26de8","Type":"ContainerStarted","Data":"c71db214edb721aea8aa75d032c94def95d0604d9ae341a7dc3e163700c4777a"}
Oct 30 13:48:16.938843 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.938816    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 13:48:16.939033 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939013    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 13:48:16.939141 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939031    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 13:48:16.939141 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939068    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 13:48:16.939141 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939123    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 13:48:16.939286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939161    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 13:48:16.939286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939165    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 13:48:16.939286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939067    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k4gxw\""
Oct 30 13:48:16.939286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939241    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:48:16.939286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939124    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 13:48:16.939470 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939342    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 13:48:16.939470 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939361    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 13:48:16.939470 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939244    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mxshj\""
Oct 30 13:48:16.939470 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939377    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 13:48:16.939470 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939403    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 13:48:16.939470 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939438    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xn5x9\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939479    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s5pv6\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939506    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939524    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5r2q6\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939564    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939567    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939504    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7j7wh\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939568    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939596    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939678    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-697m8\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939683    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939703    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939688    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939730    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 13:48:16.939766 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.939754    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 13:48:16.951552 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:16.951514    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 13:48:17.012558 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-lib-modules\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.012695 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-slash\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.012695 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-systemd\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.012695 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-cni-bin\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.012695 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012641    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-slash\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.012695 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxql\" (UniqueName: \"kubernetes.io/projected/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-kube-api-access-xpxql\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.012695 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-tuned\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.012890 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012700    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-lib-modules\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.012890 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012714    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-systemd\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.012890 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-cnibin\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.012890 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012713    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-cni-bin\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.012890 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwg4\" (UniqueName: \"kubernetes.io/projected/375bd1fd-307b-4a6b-b6be-99b492065982-kube-api-access-tdwg4\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.012890 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012769    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-cnibin\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.012890 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b3dcf54-ba8f-4080-9666-cb41a151ccad-tmp\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.012890 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2-konnectivity-ca\") pod \"konnectivity-agent-sctxm\" (UID: \"efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2\") " pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:17.012890 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-sys\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.012890 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-netns\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.012890 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-ovn\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-kubernetes\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-socket-dir-parent\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-conf-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.012992    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-sys\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysctl-d\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp28b\" (UniqueName: \"kubernetes.io/projected/0b3dcf54-ba8f-4080-9666-cb41a151ccad-kube-api-access-rp28b\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-cni-binary-copy\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-hostroot\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cni-binary-copy\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-socket-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013130    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-kubernetes\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-cni-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013147    2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 13:48:17.013308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013177    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013219    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-ovn\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013221    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013256    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-socket-dir-parent\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013271    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-netns\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013288    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013278    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-hostroot\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013311    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-conf-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.013360    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013406    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-socket-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.013427    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs podName:9acb4199-6d58-4d3d-86dc-96326517baf8 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:17.51339547 +0000 UTC m=+3.072397697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs") pod "network-metrics-daemon-l5x6g" (UID: "9acb4199-6d58-4d3d-86dc-96326517baf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013427    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysctl-d\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013547    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-cni-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvkx\" (UniqueName: \"kubernetes.io/projected/d7926b85-83e3-4e81-a403-774ea8daebe1-kube-api-access-8zvkx\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055c8a12-fcb0-440d-bacc-20dd67b2ce02-host-slash\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44fn2\" (UniqueName: \"kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2\") pod \"network-check-target-98cgg\" (UID: \"fafdd535-5a88-4207-b2a4-cc43bf43bd17\") " pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rnlm\" (UniqueName: \"kubernetes.io/projected/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-kube-api-access-2rnlm\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.014274 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-log-socket\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-cni-netd\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0bbec59-52f9-4dc6-b24b-169f666c08ae-dbus\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-hosts-file\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013819    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2-konnectivity-ca\") pod \"konnectivity-agent-sctxm\" (UID: \"efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2\") " pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/055c8a12-fcb0-440d-bacc-20dd67b2ce02-iptables-alerter-script\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-k8s-cni-cncf-io\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013912    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-k8s-cni-cncf-io\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-kubelet\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013951    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055c8a12-fcb0-440d-bacc-20dd67b2ce02-host-slash\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.013976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014017    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-cni-netd\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6hc\" (UniqueName: \"kubernetes.io/projected/08311d3a-d790-4761-96f0-e9a6b336e913-kube-api-access-9f6hc\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-run\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-var-lib-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014117    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cni-binary-copy\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.015148 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-node-log\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014153    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-cni-binary-copy\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014167    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0bbec59-52f9-4dc6-b24b-169f666c08ae-dbus\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014168    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-kubelet\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014227    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-hosts-file\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014231    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-log-socket\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014271    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-node-log\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014289    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-run\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014290    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014334    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-var-lib-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7926b85-83e3-4e81-a403-774ea8daebe1-ovn-node-metrics-cert\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-kubelet\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cnibin\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-multus-certs\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.016079 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014610    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysconfig\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-systemd\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014688    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/055c8a12-fcb0-440d-bacc-20dd67b2ce02-iptables-alerter-script\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014680    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-kubelet\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-cni-multus\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.014726    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-system-cni-dir\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014764    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cnibin\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.014776    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret podName:f0bbec59-52f9-4dc6-b24b-169f666c08ae nodeName:}" failed. No retries permitted until 2026-10-30 13:48:17.514758867 +0000 UTC m=+3.073761099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret") pod "global-pull-secret-syncer-2w5dd" (UID: "f0bbec59-52f9-4dc6-b24b-169f666c08ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014774    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-cni-multus\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014813    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-system-cni-dir\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014813    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-multus-certs\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014855    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-systemd\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014874    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysconfig\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-etc-selinux\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014921    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.016852 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-sys-fs\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-run-netns\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014972    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-sys-fs\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.014989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-var-lib-kubelet\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015003    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-etc-selinux\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015012    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-run-netns\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysctl-conf\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015058    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-var-lib-kubelet\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-system-cni-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-ovnkube-config\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-host\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-systemd-units\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015165    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysctl-conf\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfk95\" (UniqueName: \"kubernetes.io/projected/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-kube-api-access-kfk95\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015207    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-systemd-units\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-device-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015204    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-host\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.017746 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015255    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-system-cni-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015248    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-etc-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015349    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-device-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015413    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-etc-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0bbec59-52f9-4dc6-b24b-169f666c08ae-kubelet-config\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-registration-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08311d3a-d790-4761-96f0-e9a6b336e913-host\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08311d3a-d790-4761-96f0-e9a6b336e913-serviceca\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015591    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-registration-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-modprobe-d\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-daemon-config\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-etc-kubernetes\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015716    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-modprobe-d\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-env-overrides\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015764    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-ovnkube-config\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-ovnkube-script-lib\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.018286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-tmp-dir\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015828    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-etc-kubernetes\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2-agent-certs\") pod \"konnectivity-agent-sctxm\" (UID: \"efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2\") " pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k6s7\" (UniqueName: \"kubernetes.io/projected/055c8a12-fcb0-440d-bacc-20dd67b2ce02-kube-api-access-6k6s7\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz5fm\" (UniqueName: \"kubernetes.io/projected/9acb4199-6d58-4d3d-86dc-96326517baf8-kube-api-access-lz5fm\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.015967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-cni-bin\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.016138    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-cni-bin\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.016188    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-daemon-config\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.016291    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08311d3a-d790-4761-96f0-e9a6b336e913-host\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.016358    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0bbec59-52f9-4dc6-b24b-169f666c08ae-kubelet-config\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.017102    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-tmp-dir\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.017127    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-ovnkube-script-lib\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.017232    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08311d3a-d790-4761-96f0-e9a6b336e913-serviceca\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:17.018956 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.017608    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-env-overrides\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.020307 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.020284    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b3dcf54-ba8f-4080-9666-cb41a151ccad-tmp\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.022073 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.022052    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2-agent-certs\") pod \"konnectivity-agent-sctxm\" (UID: \"efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2\") " pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:17.022806 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.022773    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7926b85-83e3-4e81-a403-774ea8daebe1-ovn-node-metrics-cert\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.023316 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.023297    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-tuned\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.039330 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.039302    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwg4\" (UniqueName: \"kubernetes.io/projected/375bd1fd-307b-4a6b-b6be-99b492065982-kube-api-access-tdwg4\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.041053 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.041030    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxql\" (UniqueName: \"kubernetes.io/projected/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-kube-api-access-xpxql\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.041149 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.041033    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp28b\" (UniqueName: \"kubernetes.io/projected/0b3dcf54-ba8f-4080-9666-cb41a151ccad-kube-api-access-rp28b\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.041995 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.041868    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:48:17.041995 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.041895    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:48:17.041995 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.041911    2580 projected.go:196] Error preparing data for projected volume kube-api-access-44fn2 for pod openshift-network-diagnostics/network-check-target-98cgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:48:17.041995 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.041971    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2 podName:fafdd535-5a88-4207-b2a4-cc43bf43bd17 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:17.541952986 +0000 UTC m=+3.100955254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-44fn2" (UniqueName: "kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2") pod "network-check-target-98cgg" (UID: "fafdd535-5a88-4207-b2a4-cc43bf43bd17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:48:17.043974 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.043931    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rnlm\" (UniqueName: \"kubernetes.io/projected/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-kube-api-access-2rnlm\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.044609 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.044579    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfk95\" (UniqueName: \"kubernetes.io/projected/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-kube-api-access-kfk95\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:17.045180 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.045126    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6hc\" (UniqueName: \"kubernetes.io/projected/08311d3a-d790-4761-96f0-e9a6b336e913-kube-api-access-9f6hc\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:17.045741 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.045711    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvkx\" (UniqueName: \"kubernetes.io/projected/d7926b85-83e3-4e81-a403-774ea8daebe1-kube-api-access-8zvkx\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.045899 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.045871    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k6s7\" (UniqueName: \"kubernetes.io/projected/055c8a12-fcb0-440d-bacc-20dd67b2ce02-kube-api-access-6k6s7\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:17.047218 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.047197    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz5fm\" (UniqueName: \"kubernetes.io/projected/9acb4199-6d58-4d3d-86dc-96326517baf8-kube-api-access-lz5fm\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:17.131936 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.131909    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:48:17.133027 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.132990    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:17.142494 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.142467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:17.144916 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:17.144888    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08311d3a_d790_4761_96f0_e9a6b336e913.slice/crio-677a40a55c1c06e74ac7bcd6e73bd1403940c55684f46ee92d1754353be246a3 WatchSource:0}: Error finding container 677a40a55c1c06e74ac7bcd6e73bd1403940c55684f46ee92d1754353be246a3: Status 404 returned error can't find the container with id 677a40a55c1c06e74ac7bcd6e73bd1403940c55684f46ee92d1754353be246a3
Oct 30 13:48:17.153651 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:17.153624    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055c8a12_fcb0_440d_bacc_20dd67b2ce02.slice/crio-01204cb7fd2d72f4b3ac92a86bbb962182f4b6cb7e3dbc9d98c0e6dd9c37aaa3 WatchSource:0}: Error finding container 01204cb7fd2d72f4b3ac92a86bbb962182f4b6cb7e3dbc9d98c0e6dd9c37aaa3: Status 404 returned error can't find the container with id 01204cb7fd2d72f4b3ac92a86bbb962182f4b6cb7e3dbc9d98c0e6dd9c37aaa3
Oct 30 13:48:17.166667 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.166647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:17.167725 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.167705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:17.174814 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.174740    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:17.177890 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.177866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rjz78"
Oct 30 13:48:17.184433 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:17.184402    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb9f1c7_f4ad_4ab4_a6a0_30a6509f96f2.slice/crio-9ccc6aaf2a1f82e3bbdff57aaaa3356058de2312fd2e2a4d0f482734d0526a05 WatchSource:0}: Error finding container 9ccc6aaf2a1f82e3bbdff57aaaa3356058de2312fd2e2a4d0f482734d0526a05: Status 404 returned error can't find the container with id 9ccc6aaf2a1f82e3bbdff57aaaa3356058de2312fd2e2a4d0f482734d0526a05
Oct 30 13:48:17.184532 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.184495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:17.187006 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:17.186976    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3dcf54_ba8f_4080_9666_cb41a151ccad.slice/crio-fa7c8f0c41813f3d0eb55af1b2a6f99b329af880848a49d726a8a2341c34a72d WatchSource:0}: Error finding container fa7c8f0c41813f3d0eb55af1b2a6f99b329af880848a49d726a8a2341c34a72d: Status 404 returned error can't find the container with id fa7c8f0c41813f3d0eb55af1b2a6f99b329af880848a49d726a8a2341c34a72d
Oct 30 13:48:17.191244 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:17.191126    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7926b85_83e3_4e81_a403_774ea8daebe1.slice/crio-9364e815e66d572732f722076f92640b815f70566b1130b7142f9eaef15437c4 WatchSource:0}: Error finding container 9364e815e66d572732f722076f92640b815f70566b1130b7142f9eaef15437c4: Status 404 returned error can't find the container with id 9364e815e66d572732f722076f92640b815f70566b1130b7142f9eaef15437c4
Oct 30 13:48:17.196980 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:17.196952    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eba42f5_ebfb_457a_ba41_37ba8bed9e4c.slice/crio-5b8733a18ac8fac79d29356d289d071b29322a192622aa19a50cfd10b43522b7 WatchSource:0}: Error finding container 5b8733a18ac8fac79d29356d289d071b29322a192622aa19a50cfd10b43522b7: Status 404 returned error can't find the container with id 5b8733a18ac8fac79d29356d289d071b29322a192622aa19a50cfd10b43522b7
Oct 30 13:48:17.212004 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.211823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:17.214724 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.214697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:17.222368 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:17.222344    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37f6a9a_e81e_40a1_afd4_2726e0bf2614.slice/crio-e55e12210c8cd3e333f4f252c97f03006b1cfb7788fc7250a14725f39ef44e2d WatchSource:0}: Error finding container e55e12210c8cd3e333f4f252c97f03006b1cfb7788fc7250a14725f39ef44e2d: Status 404 returned error can't find the container with id e55e12210c8cd3e333f4f252c97f03006b1cfb7788fc7250a14725f39ef44e2d
Oct 30 13:48:17.225817 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:17.225729    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375bd1fd_307b_4a6b_b6be_99b492065982.slice/crio-c5c2a0f95a9017cab248b90d1c1dd73282463406477893e93d7dd2177e9bcc24 WatchSource:0}: Error finding container c5c2a0f95a9017cab248b90d1c1dd73282463406477893e93d7dd2177e9bcc24: Status 404 returned error can't find the container with id c5c2a0f95a9017cab248b90d1c1dd73282463406477893e93d7dd2177e9bcc24
Oct 30 13:48:17.520675 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.520636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:17.520777 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.520720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:17.520904 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.520887    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:48:17.520958 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.520952    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret podName:f0bbec59-52f9-4dc6-b24b-169f666c08ae nodeName:}" failed. No retries permitted until 2026-10-30 13:48:18.520936352 +0000 UTC m=+4.079938597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret") pod "global-pull-secret-syncer-2w5dd" (UID: "f0bbec59-52f9-4dc6-b24b-169f666c08ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:48:17.521311 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.521296    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:48:17.521357 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.521342    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs podName:9acb4199-6d58-4d3d-86dc-96326517baf8 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:18.521330405 +0000 UTC m=+4.080332631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs") pod "network-metrics-daemon-l5x6g" (UID: "9acb4199-6d58-4d3d-86dc-96326517baf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:48:17.621483 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.621453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44fn2\" (UniqueName: \"kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2\") pod \"network-check-target-98cgg\" (UID: \"fafdd535-5a88-4207-b2a4-cc43bf43bd17\") " pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:17.621742 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.621648    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:48:17.621742 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.621680    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:48:17.621742 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.621696    2580 projected.go:196] Error preparing data for projected volume kube-api-access-44fn2 for pod openshift-network-diagnostics/network-check-target-98cgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:48:17.621923 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.621762    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2 podName:fafdd535-5a88-4207-b2a4-cc43bf43bd17 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:18.621741867 +0000 UTC m=+4.180744117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-44fn2" (UniqueName: "kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2") pod "network-check-target-98cgg" (UID: "fafdd535-5a88-4207-b2a4-cc43bf43bd17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:48:17.855821 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.855755    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 13:43:17 +0000 UTC" deadline="2028-04-07 00:42:05.378704373 +0000 UTC"
Oct 30 13:48:17.855821 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.855817    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12586h53m47.522891048s"
Oct 30 13:48:17.920157 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.919923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:17.920306 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:17.920280    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:17.954551 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.954048    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" event={"ID":"e329b8d309e1f8141c44f91d4bb26de8","Type":"ContainerStarted","Data":"232b7d148291e15af4ba0fbcd6ccc0288aa4b530081ece9cdade86dcedb93dea"}
Oct 30 13:48:17.957973 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.957940    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht" event={"ID":"375bd1fd-307b-4a6b-b6be-99b492065982","Type":"ContainerStarted","Data":"c5c2a0f95a9017cab248b90d1c1dd73282463406477893e93d7dd2177e9bcc24"}
Oct 30 13:48:17.960095 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.960031    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cjg7z" event={"ID":"62977d18-4b0e-4e1e-ba64-bcda9ca2095e","Type":"ContainerStarted","Data":"1dff1ceb0266e2bc94215f7a694fb1716ab5455670d27463ec082a8364403289"}
Oct 30 13:48:17.966312 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.966248    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs" event={"ID":"d7926b85-83e3-4e81-a403-774ea8daebe1","Type":"ContainerStarted","Data":"9364e815e66d572732f722076f92640b815f70566b1130b7142f9eaef15437c4"}
Oct 30 13:48:17.972444 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.972374    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qlh2v" event={"ID":"0b3dcf54-ba8f-4080-9666-cb41a151ccad","Type":"ContainerStarted","Data":"fa7c8f0c41813f3d0eb55af1b2a6f99b329af880848a49d726a8a2341c34a72d"}
Oct 30 13:48:17.976964 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.976853    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sctxm" event={"ID":"efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2","Type":"ContainerStarted","Data":"9ccc6aaf2a1f82e3bbdff57aaaa3356058de2312fd2e2a4d0f482734d0526a05"}
Oct 30 13:48:17.981631 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.981601    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-czxww" event={"ID":"055c8a12-fcb0-440d-bacc-20dd67b2ce02","Type":"ContainerStarted","Data":"01204cb7fd2d72f4b3ac92a86bbb962182f4b6cb7e3dbc9d98c0e6dd9c37aaa3"}
Oct 30 13:48:17.993106 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.993076    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5xcz" event={"ID":"b37f6a9a-e81e-40a1-afd4-2726e0bf2614","Type":"ContainerStarted","Data":"e55e12210c8cd3e333f4f252c97f03006b1cfb7788fc7250a14725f39ef44e2d"}
Oct 30 13:48:17.999017 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:17.998923    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rjz78" event={"ID":"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c","Type":"ContainerStarted","Data":"5b8733a18ac8fac79d29356d289d071b29322a192622aa19a50cfd10b43522b7"}
Oct 30 13:48:18.001251 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:18.001180    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6pbvr" event={"ID":"08311d3a-d790-4761-96f0-e9a6b336e913","Type":"ContainerStarted","Data":"677a40a55c1c06e74ac7bcd6e73bd1403940c55684f46ee92d1754353be246a3"}
Oct 30 13:48:18.531250 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:18.531209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:18.531455 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:18.531277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:18.531455 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:18.531412    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:48:18.531663 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:18.531474    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret podName:f0bbec59-52f9-4dc6-b24b-169f666c08ae nodeName:}" failed. No retries permitted until 2026-10-30 13:48:20.531457167 +0000 UTC m=+6.090459399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret") pod "global-pull-secret-syncer-2w5dd" (UID: "f0bbec59-52f9-4dc6-b24b-169f666c08ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:48:18.531939 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:18.531910    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:48:18.532115 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:18.531977    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs podName:9acb4199-6d58-4d3d-86dc-96326517baf8 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:20.531959361 +0000 UTC m=+6.090961587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs") pod "network-metrics-daemon-l5x6g" (UID: "9acb4199-6d58-4d3d-86dc-96326517baf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:48:18.632675 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:18.632007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44fn2\" (UniqueName: \"kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2\") pod \"network-check-target-98cgg\" (UID: \"fafdd535-5a88-4207-b2a4-cc43bf43bd17\") " pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:18.632675 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:18.632188    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:48:18.632675 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:18.632209    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:48:18.632675 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:18.632232    2580 projected.go:196] Error preparing data for projected volume kube-api-access-44fn2 for pod openshift-network-diagnostics/network-check-target-98cgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:48:18.632675 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:18.632287    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2 podName:fafdd535-5a88-4207-b2a4-cc43bf43bd17 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:20.632269128 +0000 UTC m=+6.191271370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-44fn2" (UniqueName: "kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2") pod "network-check-target-98cgg" (UID: "fafdd535-5a88-4207-b2a4-cc43bf43bd17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:48:18.856984 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:18.856937    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 13:43:17 +0000 UTC" deadline="2028-03-31 15:31:57.792938322 +0000 UTC"
Oct 30 13:48:18.856984 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:18.856983    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12433h43m38.935960407s"
Oct 30 13:48:18.922229 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:18.922195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:18.922415 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:18.922327    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:18.922774 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:18.922751    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:18.922917 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:18.922856    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:19.020700 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:19.020271    2580 generic.go:356] "Generic (PLEG): container finished" podID="b30ac2208b235a46404847e29afd4a13" containerID="4d6a31e8a112d600d96c419998a74d21088ed3a05fdd7c9819e1b34a97965946" exitCode=0
Oct 30 13:48:19.020700 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:19.020366    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" event={"ID":"b30ac2208b235a46404847e29afd4a13","Type":"ContainerDied","Data":"4d6a31e8a112d600d96c419998a74d21088ed3a05fdd7c9819e1b34a97965946"}
Oct 30 13:48:19.083970 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:19.083871    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" podStartSLOduration=2.083851533 podStartE2EDuration="2.083851533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:17.995522252 +0000 UTC m=+3.554524496" watchObservedRunningTime="2026-10-30 13:48:19.083851533 +0000 UTC m=+4.642853791"
Oct 30 13:48:19.919913 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:19.919691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:19.919913 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:19.919841    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:20.035824 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:20.035785    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" event={"ID":"b30ac2208b235a46404847e29afd4a13","Type":"ContainerStarted","Data":"0be6eade63d1a73ecf40a4aa995a4038c704e5a72460395c1dd3f8dc4d570ff0"}
Oct 30 13:48:20.077804 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:20.077218    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" podStartSLOduration=3.077198756 podStartE2EDuration="3.077198756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:20.076673807 +0000 UTC m=+5.635676060" watchObservedRunningTime="2026-10-30 13:48:20.077198756 +0000 UTC m=+5.636201006"
Oct 30 13:48:20.549455 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:20.548625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:20.549455 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:20.548705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:20.549455 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:20.548846    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:48:20.549455 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:20.548914    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret podName:f0bbec59-52f9-4dc6-b24b-169f666c08ae nodeName:}" failed. No retries permitted until 2026-10-30 13:48:24.548894969 +0000 UTC m=+10.107897210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret") pod "global-pull-secret-syncer-2w5dd" (UID: "f0bbec59-52f9-4dc6-b24b-169f666c08ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:48:20.549455 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:20.549327    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:48:20.549455 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:20.549386    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs podName:9acb4199-6d58-4d3d-86dc-96326517baf8 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:24.549369305 +0000 UTC m=+10.108371555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs") pod "network-metrics-daemon-l5x6g" (UID: "9acb4199-6d58-4d3d-86dc-96326517baf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:48:20.649748 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:20.649706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44fn2\" (UniqueName: \"kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2\") pod \"network-check-target-98cgg\" (UID: \"fafdd535-5a88-4207-b2a4-cc43bf43bd17\") " pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:20.649985 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:20.649943    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:48:20.649985 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:20.649964    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:48:20.649985 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:20.649979    2580 projected.go:196] Error preparing data for projected volume kube-api-access-44fn2 for pod openshift-network-diagnostics/network-check-target-98cgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:48:20.650152 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:20.650040    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2 podName:fafdd535-5a88-4207-b2a4-cc43bf43bd17 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:24.650020922 +0000 UTC m=+10.209023152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-44fn2" (UniqueName: "kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2") pod "network-check-target-98cgg" (UID: "fafdd535-5a88-4207-b2a4-cc43bf43bd17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:48:20.924468 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:20.922741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:20.924468 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:20.922877    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:20.924468 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:20.923364    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:20.924468 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:20.923467    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:21.054878 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:21.054832    2580 generic.go:356] "Generic (PLEG): container finished" podID="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" containerID="4caa023f224a846f660c150030e1463a156e1ffd286672fcda8dc46235e67cec" exitCode=0
Oct 30 13:48:21.055891 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:21.055453    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5xcz" event={"ID":"b37f6a9a-e81e-40a1-afd4-2726e0bf2614","Type":"ContainerDied","Data":"4caa023f224a846f660c150030e1463a156e1ffd286672fcda8dc46235e67cec"}
Oct 30 13:48:21.919464 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:21.919423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:21.919671 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:21.919607    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:22.061739 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:22.061698    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sctxm" event={"ID":"efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2","Type":"ContainerStarted","Data":"de4b24897a6c1107d29b6d770cedf56acc6b3f0b0e2d2ba55b9d19a69482cce6"}
Oct 30 13:48:22.923635 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:22.922891    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:22.923635 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:22.923032    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:22.923635 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:22.923450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:22.923635 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:22.923554    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:23.072948 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:23.072510    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6pbvr" event={"ID":"08311d3a-d790-4761-96f0-e9a6b336e913","Type":"ContainerStarted","Data":"9d0805b776621161f9d65bca6bfbef602f38c8a9723bfbe18dce2d88f531cb6f"}
Oct 30 13:48:23.130057 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:23.129777    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sctxm" podStartSLOduration=4.1619529 podStartE2EDuration="8.129756979s" totalImagesPullingTime="3.967804079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:22.103510551 +0000 UTC m=+7.662512803" watchObservedRunningTime="2026-10-30 13:48:23.129756979 +0000 UTC m=+8.688759231"
Oct 30 13:48:23.920026 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:23.919993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:23.920199 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:23.920146    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:24.081308 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:24.081270    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht" event={"ID":"375bd1fd-307b-4a6b-b6be-99b492065982","Type":"ContainerStarted","Data":"9c95227526833cb1786e42da1b5a235185065d9e7b331d915f06217ec8419fca"}
Oct 30 13:48:24.584888 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:24.583943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:24.584888 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:24.584035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:24.584888 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:24.584178    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:48:24.584888 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:24.584249    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret podName:f0bbec59-52f9-4dc6-b24b-169f666c08ae nodeName:}" failed. No retries permitted until 2026-10-30 13:48:32.58422835 +0000 UTC m=+18.143230594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret") pod "global-pull-secret-syncer-2w5dd" (UID: "f0bbec59-52f9-4dc6-b24b-169f666c08ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:48:24.584888 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:24.584784    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:48:24.584888 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:24.584845    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs podName:9acb4199-6d58-4d3d-86dc-96326517baf8 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:32.584828403 +0000 UTC m=+18.143830633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs") pod "network-metrics-daemon-l5x6g" (UID: "9acb4199-6d58-4d3d-86dc-96326517baf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:48:24.687850 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:24.684475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44fn2\" (UniqueName: \"kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2\") pod \"network-check-target-98cgg\" (UID: \"fafdd535-5a88-4207-b2a4-cc43bf43bd17\") " pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:24.687850 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:24.684675    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:48:24.687850 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:24.684695    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:48:24.687850 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:24.684708    2580 projected.go:196] Error preparing data for projected volume kube-api-access-44fn2 for pod openshift-network-diagnostics/network-check-target-98cgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:48:24.687850 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:24.684766    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2 podName:fafdd535-5a88-4207-b2a4-cc43bf43bd17 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:32.684746161 +0000 UTC m=+18.243748404 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-44fn2" (UniqueName: "kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2") pod "network-check-target-98cgg" (UID: "fafdd535-5a88-4207-b2a4-cc43bf43bd17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:48:24.854030 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:24.853887    2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 13:48:24.920833 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:24.920797    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:24.921040 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:24.920912    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:24.921040 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:24.920800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:24.921167 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:24.921045    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:25.087937 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:25.087856    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht" event={"ID":"375bd1fd-307b-4a6b-b6be-99b492065982","Type":"ContainerStarted","Data":"66adfd1d8a0bf5706a4af2406e5677833a0a245e8126d49627318492f365bd6c"}
Oct 30 13:48:25.168605 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:25.168562    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:25.851688 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:25.851657    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:25.852241 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:25.852217    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:25.853702 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:25.853489    2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T13:48:24.853907825Z","UUID":"514da5e8-ef41-4d9d-a72b-ef0171791375","Handler":null,"Name":"","Endpoint":""}
Oct 30 13:48:25.855609 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:25.855207    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 13:48:25.855609 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:25.855242    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 13:48:25.896028 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:25.894912    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6pbvr" podStartSLOduration=5.818832026 podStartE2EDuration="10.894891547s" totalImagesPullingTime="5.076059521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:23.130656656 +0000 UTC m=+8.689658910" watchObservedRunningTime="2026-10-30 13:48:25.894891547 +0000 UTC m=+11.453893800"
Oct 30 13:48:25.920002 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:25.919969    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:25.920181 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:25.920104    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:26.092315 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:26.092280    2580 generic.go:356] "Generic (PLEG): container finished" podID="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" containerID="fe143a67765015f2c6fd5669cb3b3dc00cb0852f63b5a53c64bec6d18328478f" exitCode=0
Oct 30 13:48:26.092777 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:26.092367    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5xcz" event={"ID":"b37f6a9a-e81e-40a1-afd4-2726e0bf2614","Type":"ContainerDied","Data":"fe143a67765015f2c6fd5669cb3b3dc00cb0852f63b5a53c64bec6d18328478f"}
Oct 30 13:48:26.097003 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:26.096452    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rjz78" event={"ID":"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c","Type":"ContainerStarted","Data":"9369ed02ca11124bfbd35c84cb6a487770be204c8ae22259b05725f41a82b491"}
Oct 30 13:48:26.101081 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:26.100834    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cjg7z" event={"ID":"62977d18-4b0e-4e1e-ba64-bcda9ca2095e","Type":"ContainerStarted","Data":"a63dfa1119926cbcb0c61898557c42b81429f11a27de3e8fb374a8cc276f3590"}
Oct 30 13:48:26.101730 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:26.101681    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:26.273222 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:26.272904    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cjg7z" podStartSLOduration=2.830040935 podStartE2EDuration="11.272886074s" totalImagesPullingTime="8.442845139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:26.226407725 +0000 UTC m=+11.785409975" watchObservedRunningTime="2026-10-30 13:48:26.272886074 +0000 UTC m=+11.831888327"
Oct 30 13:48:26.273538 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:26.273498    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rjz78" podStartSLOduration=3.315172231 podStartE2EDuration="11.273488912s" totalImagesPullingTime="7.958316681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:26.272493838 +0000 UTC m=+11.831496089" watchObservedRunningTime="2026-10-30 13:48:26.273488912 +0000 UTC m=+11.832491164"
Oct 30 13:48:26.919997 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:26.919965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:26.920150 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:26.920014    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:26.920150 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:26.920125    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:26.920283 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:26.920247    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:27.106527 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:27.106484    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht" event={"ID":"375bd1fd-307b-4a6b-b6be-99b492065982","Type":"ContainerStarted","Data":"c073fe47832ce1be4bd90b3813483e009fe24fac61b36cd4cde8e7eb5bb31b7f"}
Oct 30 13:48:27.108874 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:27.108838    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-czxww" event={"ID":"055c8a12-fcb0-440d-bacc-20dd67b2ce02","Type":"ContainerStarted","Data":"12558e56b7fc945c046eba6fb6bb39d32b37b03d59ffd03b4dd7575f70de762d"}
Oct 30 13:48:27.112648 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:27.111774    2580 generic.go:356] "Generic (PLEG): container finished" podID="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" containerID="bffce27f46f1de025fa3e5d3b9ec636a60b56437f356200209ead96b6e5d07aa" exitCode=0
Oct 30 13:48:27.112648 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:27.112022    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5xcz" event={"ID":"b37f6a9a-e81e-40a1-afd4-2726e0bf2614","Type":"ContainerDied","Data":"bffce27f46f1de025fa3e5d3b9ec636a60b56437f356200209ead96b6e5d07aa"}
Oct 30 13:48:27.153219 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:27.153162    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht" podStartSLOduration=3.718905833 podStartE2EDuration="12.153143814s" totalImagesPullingTime="8.434237981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:27.152520353 +0000 UTC m=+12.711522606" watchObservedRunningTime="2026-10-30 13:48:27.153143814 +0000 UTC m=+12.712146065"
Oct 30 13:48:27.194413 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:27.194355    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-czxww" podStartSLOduration=3.864083355 podStartE2EDuration="12.194334275s" totalImagesPullingTime="8.33025092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:27.193710824 +0000 UTC m=+12.752713076" watchObservedRunningTime="2026-10-30 13:48:27.194334275 +0000 UTC m=+12.753336530"
Oct 30 13:48:27.920149 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:27.920120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:27.920315 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:27.920248    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:28.115348 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:28.115324    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qlh2v" event={"ID":"0b3dcf54-ba8f-4080-9666-cb41a151ccad","Type":"ContainerStarted","Data":"4aeabc93bf3e785acbf004694a1390627df2428d53e69c2c607bbbdd04811190"}
Oct 30 13:48:28.157969 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:28.157917    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qlh2v" podStartSLOduration=2.646938785 podStartE2EDuration="13.157898184s" totalImagesPullingTime="10.510959399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:28.157852129 +0000 UTC m=+13.716854379" watchObservedRunningTime="2026-10-30 13:48:28.157898184 +0000 UTC m=+13.716900423"
Oct 30 13:48:28.919794 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:28.919762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:28.919953 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:28.919770    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:28.919953 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:28.919879    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:28.920076 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:28.919977    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:29.119646 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:29.119612    2580 generic.go:356] "Generic (PLEG): container finished" podID="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" containerID="2cc7ab8e574457a5d425090c0db23fe0cd349eaf67ce8bcaf5039ee810a13366" exitCode=0
Oct 30 13:48:29.120096 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:29.119665    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5xcz" event={"ID":"b37f6a9a-e81e-40a1-afd4-2726e0bf2614","Type":"ContainerDied","Data":"2cc7ab8e574457a5d425090c0db23fe0cd349eaf67ce8bcaf5039ee810a13366"}
Oct 30 13:48:29.920239 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:29.920206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:29.920401 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:29.920339    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:30.920130 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:30.919938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:30.920736 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:30.919956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:30.920736 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:30.920242    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:30.920736 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:30.920319    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:31.126853 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:31.126752    2580 generic.go:356] "Generic (PLEG): container finished" podID="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" containerID="8767c7145b2b6cd50a2dd8d1280bc829e3e73575745798fe842253ae52476491" exitCode=0
Oct 30 13:48:31.126853 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:31.126816    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5xcz" event={"ID":"b37f6a9a-e81e-40a1-afd4-2726e0bf2614","Type":"ContainerDied","Data":"8767c7145b2b6cd50a2dd8d1280bc829e3e73575745798fe842253ae52476491"}
Oct 30 13:48:31.920129 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:31.920095    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:31.920485 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:31.920200    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:32.132031 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:32.132000    2580 generic.go:356] "Generic (PLEG): container finished" podID="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" containerID="59a86eb5be06241c11f43f93b7fb262c303faacd8e0b3789c8f4278f6f6100e8" exitCode=0
Oct 30 13:48:32.132152 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:32.132044    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5xcz" event={"ID":"b37f6a9a-e81e-40a1-afd4-2726e0bf2614","Type":"ContainerDied","Data":"59a86eb5be06241c11f43f93b7fb262c303faacd8e0b3789c8f4278f6f6100e8"}
Oct 30 13:48:32.641060 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:32.641033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:32.641165 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:32.641093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:32.641323 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:32.641170    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:48:32.641323 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:32.641198    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:48:32.641323 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:32.641237    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs podName:9acb4199-6d58-4d3d-86dc-96326517baf8 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:48.641218184 +0000 UTC m=+34.200220412 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs") pod "network-metrics-daemon-l5x6g" (UID: "9acb4199-6d58-4d3d-86dc-96326517baf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:48:32.641323 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:32.641256    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret podName:f0bbec59-52f9-4dc6-b24b-169f666c08ae nodeName:}" failed. No retries permitted until 2026-10-30 13:48:48.641246472 +0000 UTC m=+34.200248700 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret") pod "global-pull-secret-syncer-2w5dd" (UID: "f0bbec59-52f9-4dc6-b24b-169f666c08ae") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:48:32.742152 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:32.742088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44fn2\" (UniqueName: \"kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2\") pod \"network-check-target-98cgg\" (UID: \"fafdd535-5a88-4207-b2a4-cc43bf43bd17\") " pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:32.742278 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:32.742253    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:48:32.742278 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:32.742275    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:48:32.742377 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:32.742288    2580 projected.go:196] Error preparing data for projected volume kube-api-access-44fn2 for pod openshift-network-diagnostics/network-check-target-98cgg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:48:32.742377 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:32.742347    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2 podName:fafdd535-5a88-4207-b2a4-cc43bf43bd17 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:48.742328037 +0000 UTC m=+34.301330277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-44fn2" (UniqueName: "kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2") pod "network-check-target-98cgg" (UID: "fafdd535-5a88-4207-b2a4-cc43bf43bd17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:48:32.920065 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:32.920032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:32.920182 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:32.920032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:32.920182 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:32.920134    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:32.920497 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:32.920199    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:33.136768 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:33.136741    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g5xcz" event={"ID":"b37f6a9a-e81e-40a1-afd4-2726e0bf2614","Type":"ContainerStarted","Data":"6c6b954b7c000953c92fb9bfdb131932274be39be00a2315dabfc98dabb1a8b4"}
Oct 30 13:48:33.139335 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:33.139316    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs" event={"ID":"d7926b85-83e3-4e81-a403-774ea8daebe1","Type":"ContainerStarted","Data":"d58cef2d19ffdaadf3d2ec96567a4df925b7f00099789bcbd64139077bcf6532"}
Oct 30 13:48:33.139435 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:33.139367    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs" event={"ID":"d7926b85-83e3-4e81-a403-774ea8daebe1","Type":"ContainerStarted","Data":"a68fc49cfd3138ac3d89c97ee225a0b7d7dbca6cd675bc13422b61848fa966e8"}
Oct 30 13:48:33.139435 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:33.139397    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs" event={"ID":"d7926b85-83e3-4e81-a403-774ea8daebe1","Type":"ContainerStarted","Data":"ee49e977df656438f66bda634fee9b8f80e040e3238034ed0b3b65d4d884e4a3"}
Oct 30 13:48:33.139435 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:33.139426    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs" event={"ID":"d7926b85-83e3-4e81-a403-774ea8daebe1","Type":"ContainerStarted","Data":"d11cb29bb7ca06731790b0f5edababd778edd46c9a59a82669e6c4b86a3f39cb"}
Oct 30 13:48:33.139530 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:33.139452    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs" event={"ID":"d7926b85-83e3-4e81-a403-774ea8daebe1","Type":"ContainerStarted","Data":"66e8cb5c3bd63f95194ff2e7ca9e250215b28694a268bb1d012c6a8133e7d833"}
Oct 30 13:48:33.139530 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:33.139477    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs" event={"ID":"d7926b85-83e3-4e81-a403-774ea8daebe1","Type":"ContainerStarted","Data":"f7b7fc5df286de61e547863ef6fe54c478ab044fd85d3e81abc844ed99a0f733"}
Oct 30 13:48:33.187233 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:33.187185    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g5xcz" podStartSLOduration=7.59530933 podStartE2EDuration="18.187169738s" totalImagesPullingTime="10.591860408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:33.18611045 +0000 UTC m=+18.745112699" watchObservedRunningTime="2026-10-30 13:48:33.187169738 +0000 UTC m=+18.746171986"
Oct 30 13:48:33.919908 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:33.919882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:33.920037 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:33.919974    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:34.920603 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:34.920574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:34.920995 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:34.920655    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:34.920995 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:34.920718    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:34.920995 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:34.920826    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:35.145438 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:35.145409    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs" event={"ID":"d7926b85-83e3-4e81-a403-774ea8daebe1","Type":"ContainerStarted","Data":"deaa3320cbedc14fb411b5c3559060110cca62247cff012db5b5e9f23bf92d83"}
Oct 30 13:48:35.919438 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:35.919408    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:35.919585 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:35.919529    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:36.919843 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:36.919817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:36.919843 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:36.919844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:36.920377 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:36.919912    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:36.920377 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:36.920028    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:37.153647 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:37.153609    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs" event={"ID":"d7926b85-83e3-4e81-a403-774ea8daebe1","Type":"ContainerStarted","Data":"15cfb55b576884883337c6972282612fba6cbcb5ea18412210e4247cbaa27dc3"}
Oct 30 13:48:37.153937 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:37.153919    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:37.168592 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:37.168571    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:37.241740 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:37.241698    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs" podStartSLOduration=7.045195108 podStartE2EDuration="22.241687451s" totalImagesPullingTime="15.196492343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:37.241190305 +0000 UTC m=+22.800192551" watchObservedRunningTime="2026-10-30 13:48:37.241687451 +0000 UTC m=+22.800689698"
Oct 30 13:48:37.920319 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:37.920296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:37.920704 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:37.920390    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:38.155622 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:38.155593    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:38.155752 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:38.155662    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:38.169791 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:38.169768    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:38.863879 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:38.863292    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-98cgg"]
Oct 30 13:48:38.863879 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:38.863452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:38.863879 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:38.863656    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:38.866158 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:38.865894    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l5x6g"]
Oct 30 13:48:38.866158 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:38.865998    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2w5dd"]
Oct 30 13:48:38.866158 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:38.866085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:38.866348 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:38.866211    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:38.866348 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:38.866298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:38.866436 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:38.866409    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:40.920211 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:40.920184    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:40.920761 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:40.920189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:40.920761 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:40.920317    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:40.920761 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:40.920307    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:40.920761 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:40.920391    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:40.920761 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:40.920449    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:42.919822 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:42.919795    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:42.919822 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:42.919822    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:42.920342 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:42.919801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:42.920342 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:42.919930    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w5dd" podUID="f0bbec59-52f9-4dc6-b24b-169f666c08ae"
Oct 30 13:48:42.920342 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:42.920025    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5x6g" podUID="9acb4199-6d58-4d3d-86dc-96326517baf8"
Oct 30 13:48:42.920342 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:42.920111    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cgg" podUID="fafdd535-5a88-4207-b2a4-cc43bf43bd17"
Oct 30 13:48:44.721192 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.721165    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeReady"
Oct 30 13:48:44.721618 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.721287    2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 13:48:44.788660 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.788631    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xpz2s"]
Oct 30 13:48:44.794830 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.794802    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xpz2s"
Oct 30 13:48:44.809803 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.809777    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xpz2s"]
Oct 30 13:48:44.854645 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.854583    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nr98w"]
Oct 30 13:48:44.857846 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.857828    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:44.876072 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.876047    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nr98w"]
Oct 30 13:48:44.904392 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.904369    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 13:48:44.904392 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.904379    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 13:48:44.904567 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.904415    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 13:48:44.904567 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.904440    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 13:48:44.904567 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.904495    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b7cls\""
Oct 30 13:48:44.904567 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.904535    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 13:48:44.904692 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.904602    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-snt8q\""
Oct 30 13:48:44.920637 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.920617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:44.920784 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.920762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:44.920929 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.920911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:44.933476 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.933410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4819261-bdd9-419e-abd3-15fa171de861-tmp-dir\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:44.933595 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.933479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkn6d\" (UniqueName: \"kubernetes.io/projected/b4819261-bdd9-419e-abd3-15fa171de861-kube-api-access-dkn6d\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:44.933595 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.933506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlxg8\" (UniqueName: \"kubernetes.io/projected/6bd2311e-929b-4d5a-9857-47c0500c3be1-kube-api-access-qlxg8\") pod \"ingress-canary-xpz2s\" (UID: \"6bd2311e-929b-4d5a-9857-47c0500c3be1\") " pod="openshift-ingress-canary/ingress-canary-xpz2s"
Oct 30 13:48:44.933595 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.933529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4819261-bdd9-419e-abd3-15fa171de861-config-volume\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:44.933744 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.933616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bd2311e-929b-4d5a-9857-47c0500c3be1-cert\") pod \"ingress-canary-xpz2s\" (UID: \"6bd2311e-929b-4d5a-9857-47c0500c3be1\") " pod="openshift-ingress-canary/ingress-canary-xpz2s"
Oct 30 13:48:44.933744 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:44.933711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:45.000314 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.000287    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 13:48:45.000314 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.000285    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 13:48:45.000521 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.000347    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 13:48:45.000521 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.000368    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gw59s\""
Oct 30 13:48:45.000521 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.000520    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w8shz\""
Oct 30 13:48:45.000662 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.000523    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-smj8j\""
Oct 30 13:48:45.000662 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.000590    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 13:48:45.034399 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.034379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4819261-bdd9-419e-abd3-15fa171de861-tmp-dir\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:45.034514 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.034417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkn6d\" (UniqueName: \"kubernetes.io/projected/b4819261-bdd9-419e-abd3-15fa171de861-kube-api-access-dkn6d\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:45.034514 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.034434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxg8\" (UniqueName: \"kubernetes.io/projected/6bd2311e-929b-4d5a-9857-47c0500c3be1-kube-api-access-qlxg8\") pod \"ingress-canary-xpz2s\" (UID: \"6bd2311e-929b-4d5a-9857-47c0500c3be1\") " pod="openshift-ingress-canary/ingress-canary-xpz2s"
Oct 30 13:48:45.034514 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.034451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4819261-bdd9-419e-abd3-15fa171de861-config-volume\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:45.034514 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.034470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bd2311e-929b-4d5a-9857-47c0500c3be1-cert\") pod \"ingress-canary-xpz2s\" (UID: \"6bd2311e-929b-4d5a-9857-47c0500c3be1\") " pod="openshift-ingress-canary/ingress-canary-xpz2s"
Oct 30 13:48:45.034743 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.034671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:45.034796 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.034745    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4819261-bdd9-419e-abd3-15fa171de861-tmp-dir\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:45.034844 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:45.034816    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 13:48:45.034896 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:45.034870    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls podName:b4819261-bdd9-419e-abd3-15fa171de861 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:45.534852189 +0000 UTC m=+31.093854415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls") pod "dns-default-nr98w" (UID: "b4819261-bdd9-419e-abd3-15fa171de861") : secret "dns-default-metrics-tls" not found
Oct 30 13:48:45.035077 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.035056    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4819261-bdd9-419e-abd3-15fa171de861-config-volume\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:45.037664 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.037647    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bd2311e-929b-4d5a-9857-47c0500c3be1-cert\") pod \"ingress-canary-xpz2s\" (UID: \"6bd2311e-929b-4d5a-9857-47c0500c3be1\") " pod="openshift-ingress-canary/ingress-canary-xpz2s"
Oct 30 13:48:45.061025 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.061001    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxg8\" (UniqueName: \"kubernetes.io/projected/6bd2311e-929b-4d5a-9857-47c0500c3be1-kube-api-access-qlxg8\") pod \"ingress-canary-xpz2s\" (UID: \"6bd2311e-929b-4d5a-9857-47c0500c3be1\") " pod="openshift-ingress-canary/ingress-canary-xpz2s"
Oct 30 13:48:45.061339 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.061322    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkn6d\" (UniqueName: \"kubernetes.io/projected/b4819261-bdd9-419e-abd3-15fa171de861-kube-api-access-dkn6d\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:45.106411 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.106361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xpz2s"
Oct 30 13:48:45.307382 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.307352    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pb77c"]
Oct 30 13:48:45.333979 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.333955    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pb77c"]
Oct 30 13:48:45.334085 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.334053    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.347697 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.347677    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-f4lw9-sg475"]
Oct 30 13:48:45.385467 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.385419    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f4lw9-sg475"]
Oct 30 13:48:45.385575 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.385500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.397848 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.397832    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 13:48:45.397973 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.397958    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w6ztc\""
Oct 30 13:48:45.437319 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.437294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.437427 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.437327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tz4h\" (UniqueName: \"kubernetes.io/projected/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-api-access-2tz4h\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.437427 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.437356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-crio-socket\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.437554 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.437519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-data-volume\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.437613 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.437566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.437671 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.437622    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 13:48:45.437671 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.437630    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 13:48:45.437671 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.437642    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 13:48:45.437785 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.437687    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 13:48:45.450146 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.450126    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 13:48:45.467112 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.467090    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xpz2s"]
Oct 30 13:48:45.470564 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:45.470523    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd2311e_929b_4d5a_9857_47c0500c3be1.slice/crio-74d518a74273128a954146e126cbc81fcdf372a21e04a3ee7a777d3e8840a907 WatchSource:0}: Error finding container 74d518a74273128a954146e126cbc81fcdf372a21e04a3ee7a777d3e8840a907: Status 404 returned error can't find the container with id 74d518a74273128a954146e126cbc81fcdf372a21e04a3ee7a777d3e8840a907
Oct 30 13:48:45.498526 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.498502    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d6wsl\""
Oct 30 13:48:45.537811 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.537791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-crio-socket\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.537902 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.537818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-service-ca-bundle\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.537902 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.537864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-data-volume\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.537902 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.537880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.538044 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.537916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-trusted-ca-bundle\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.538044 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.537940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swft4\" (UniqueName: \"kubernetes.io/projected/c6f44206-dd53-4dae-b68c-84a8317227b5-kube-api-access-swft4\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.538044 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:45.537965    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:45.538044 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:45.538006    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.037993848 +0000 UTC m=+31.596996093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:45.538044 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.538004    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-crio-socket\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.538286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.538125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:45.538286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.538157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.538286 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.538163    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-data-volume\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.538286 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:45.538245    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 13:48:45.538286 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:45.538269    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 13:48:45.538585 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:45.538308    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.038293082 +0000 UTC m=+31.597295332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : configmap "kube-rbac-proxy" not found
Oct 30 13:48:45.538585 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.538327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tz4h\" (UniqueName: \"kubernetes.io/projected/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-api-access-2tz4h\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.538585 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.538356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6f44206-dd53-4dae-b68c-84a8317227b5-archives-path\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.538585 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:45.538378    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls podName:b4819261-bdd9-419e-abd3-15fa171de861 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.538363496 +0000 UTC m=+32.097365735 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls") pod "dns-default-nr98w" (UID: "b4819261-bdd9-419e-abd3-15fa171de861") : secret "dns-default-metrics-tls" not found
Oct 30 13:48:45.567632 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.567611    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tz4h\" (UniqueName: \"kubernetes.io/projected/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-api-access-2tz4h\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:45.639498 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.639449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-service-ca-bundle\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.639596 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.639509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-trusted-ca-bundle\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.639596 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.639525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swft4\" (UniqueName: \"kubernetes.io/projected/c6f44206-dd53-4dae-b68c-84a8317227b5-kube-api-access-swft4\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.639596 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.639577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6f44206-dd53-4dae-b68c-84a8317227b5-archives-path\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.640002 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.639987    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6f44206-dd53-4dae-b68c-84a8317227b5-archives-path\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.640106 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.640090    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-service-ca-bundle\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.640577 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.640558    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-trusted-ca-bundle\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.665717 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.665697    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swft4\" (UniqueName: \"kubernetes.io/projected/c6f44206-dd53-4dae-b68c-84a8317227b5-kube-api-access-swft4\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.715633 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.715618    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:45.989756 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:45.989691    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9cs6t"]
Oct 30 13:48:46.010587 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.010556    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9cs6t"]
Oct 30 13:48:46.010793 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.010614    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:46.023367 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.023335    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f4lw9-sg475"]
Oct 30 13:48:46.027092 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.026968    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 13:48:46.027092 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.027013    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 13:48:46.027231 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.027111    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 13:48:46.027231 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.027208    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 13:48:46.027401 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.027354    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-825mf\""
Oct 30 13:48:46.029272 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:46.029242    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f44206_dd53_4dae_b68c_84a8317227b5.slice/crio-7e7b183942858b307e989212f04f7088a104787ee77e63ff40852eb888bc702d WatchSource:0}: Error finding container 7e7b183942858b307e989212f04f7088a104787ee77e63ff40852eb888bc702d: Status 404 returned error can't find the container with id 7e7b183942858b307e989212f04f7088a104787ee77e63ff40852eb888bc702d
Oct 30 13:48:46.042405 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.042381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:46.042510 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.042449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:46.042575 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:46.042525    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 13:48:46.042613 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:46.042596    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:47.042579105 +0000 UTC m=+32.601581329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : configmap "kube-rbac-proxy" not found
Oct 30 13:48:46.042613 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:46.042529    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:46.042686 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:46.042631    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:47.042624456 +0000 UTC m=+32.601626723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:46.142803 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.142776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vcdc\" (UniqueName: \"kubernetes.io/projected/bbc411f7-86c3-4d69-8f20-af096604e14a-kube-api-access-5vcdc\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:46.142936 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.142832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bbc411f7-86c3-4d69-8f20-af096604e14a-signing-key\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:46.142979 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.142960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bbc411f7-86c3-4d69-8f20-af096604e14a-signing-cabundle\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:46.173147 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.173120    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4lw9-sg475" event={"ID":"c6f44206-dd53-4dae-b68c-84a8317227b5","Type":"ContainerStarted","Data":"7e7b183942858b307e989212f04f7088a104787ee77e63ff40852eb888bc702d"}
Oct 30 13:48:46.175294 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.175264    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xpz2s" event={"ID":"6bd2311e-929b-4d5a-9857-47c0500c3be1","Type":"ContainerStarted","Data":"74d518a74273128a954146e126cbc81fcdf372a21e04a3ee7a777d3e8840a907"}
Oct 30 13:48:46.243886 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.243828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bbc411f7-86c3-4d69-8f20-af096604e14a-signing-cabundle\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:46.243886 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.243860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vcdc\" (UniqueName: \"kubernetes.io/projected/bbc411f7-86c3-4d69-8f20-af096604e14a-kube-api-access-5vcdc\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:46.244070 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.243906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bbc411f7-86c3-4d69-8f20-af096604e14a-signing-key\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:46.244534 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.244509    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bbc411f7-86c3-4d69-8f20-af096604e14a-signing-cabundle\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:46.246463 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.246440    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bbc411f7-86c3-4d69-8f20-af096604e14a-signing-key\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:46.268198 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.268171    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vcdc\" (UniqueName: \"kubernetes.io/projected/bbc411f7-86c3-4d69-8f20-af096604e14a-kube-api-access-5vcdc\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:46.343405 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.343373    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:46.546754 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.546670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:46.546971 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:46.546927    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 13:48:46.547033 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:46.547000    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls podName:b4819261-bdd9-419e-abd3-15fa171de861 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:48.546981463 +0000 UTC m=+34.105983694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls") pod "dns-default-nr98w" (UID: "b4819261-bdd9-419e-abd3-15fa171de861") : secret "dns-default-metrics-tls" not found
Oct 30 13:48:46.679355 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:46.679322    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9cs6t"]
Oct 30 13:48:46.683773 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:46.683742    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc411f7_86c3_4d69_8f20_af096604e14a.slice/crio-dd5bf80e3d7cf6dabf691d654b4a105f0793c63e825814348f28e3c752dc9222 WatchSource:0}: Error finding container dd5bf80e3d7cf6dabf691d654b4a105f0793c63e825814348f28e3c752dc9222: Status 404 returned error can't find the container with id dd5bf80e3d7cf6dabf691d654b4a105f0793c63e825814348f28e3c752dc9222
Oct 30 13:48:47.051332 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:47.051291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:47.051801 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:47.051368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:47.051801 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:47.051464    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 13:48:47.051801 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:47.051522    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:49.051504356 +0000 UTC m=+34.610506584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : configmap "kube-rbac-proxy" not found
Oct 30 13:48:47.051917 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:47.051858    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:47.051917 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:47.051897    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:49.051886523 +0000 UTC m=+34.610888748 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:47.179373 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:47.179338    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t" event={"ID":"bbc411f7-86c3-4d69-8f20-af096604e14a","Type":"ContainerStarted","Data":"dd5bf80e3d7cf6dabf691d654b4a105f0793c63e825814348f28e3c752dc9222"}
Oct 30 13:48:48.563587 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:48.563554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:48.563937 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:48.563699    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 13:48:48.563937 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:48.563758    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls podName:b4819261-bdd9-419e-abd3-15fa171de861 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:52.563744436 +0000 UTC m=+38.122746665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls") pod "dns-default-nr98w" (UID: "b4819261-bdd9-419e-abd3-15fa171de861") : secret "dns-default-metrics-tls" not found
Oct 30 13:48:48.664680 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:48.664643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:48.664819 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:48.664747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:48.664893 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:48.664864    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 13:48:48.664988 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:48.664923    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs podName:9acb4199-6d58-4d3d-86dc-96326517baf8 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:20.664907262 +0000 UTC m=+66.223909500 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs") pod "network-metrics-daemon-l5x6g" (UID: "9acb4199-6d58-4d3d-86dc-96326517baf8") : secret "metrics-daemon-secret" not found
Oct 30 13:48:48.667040 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:48.667013    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:48.765906 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:48.765881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44fn2\" (UniqueName: \"kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2\") pod \"network-check-target-98cgg\" (UID: \"fafdd535-5a88-4207-b2a4-cc43bf43bd17\") " pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:48.768324 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:48.768306    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44fn2\" (UniqueName: \"kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2\") pod \"network-check-target-98cgg\" (UID: \"fafdd535-5a88-4207-b2a4-cc43bf43bd17\") " pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:48.837733 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:48.837658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:48.841134 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:48.841107    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:49.069714 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:49.069680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:49.069887 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:49.069745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:49.069887 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:49.069813    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 13:48:49.069887 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:49.069848    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:49.070037 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:49.069893    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:53.06987556 +0000 UTC m=+38.628877805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : configmap "kube-rbac-proxy" not found
Oct 30 13:48:49.070037 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:49.069909    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:53.069903353 +0000 UTC m=+38.628905578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:49.161467 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:49.161381    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-98cgg"]
Oct 30 13:48:49.168876 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:49.168850    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafdd535_5a88_4207_b2a4_cc43bf43bd17.slice/crio-7031183246a934e66f3178646e691d7ad66cdb00d8b9eb727fadfd003e8e90e3 WatchSource:0}: Error finding container 7031183246a934e66f3178646e691d7ad66cdb00d8b9eb727fadfd003e8e90e3: Status 404 returned error can't find the container with id 7031183246a934e66f3178646e691d7ad66cdb00d8b9eb727fadfd003e8e90e3
Oct 30 13:48:49.187998 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:49.187970    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4lw9-sg475" event={"ID":"c6f44206-dd53-4dae-b68c-84a8317227b5","Type":"ContainerStarted","Data":"07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258"}
Oct 30 13:48:49.190213 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:49.190184    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xpz2s" event={"ID":"6bd2311e-929b-4d5a-9857-47c0500c3be1","Type":"ContainerStarted","Data":"c601ef2ed09d79371405e663b91467455e75a77b3731580d086e0271a7fa5ddd"}
Oct 30 13:48:49.191981 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:49.191956    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-98cgg" event={"ID":"fafdd535-5a88-4207-b2a4-cc43bf43bd17","Type":"ContainerStarted","Data":"7031183246a934e66f3178646e691d7ad66cdb00d8b9eb727fadfd003e8e90e3"}
Oct 30 13:48:49.206993 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:49.206934    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2w5dd"]
Oct 30 13:48:49.211796 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:49.211770    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bbec59_52f9_4dc6_b24b_169f666c08ae.slice/crio-b96ada9d2bf66950da4af8a81e33b90def208c4fcf20839770dda4172c62bb37 WatchSource:0}: Error finding container b96ada9d2bf66950da4af8a81e33b90def208c4fcf20839770dda4172c62bb37: Status 404 returned error can't find the container with id b96ada9d2bf66950da4af8a81e33b90def208c4fcf20839770dda4172c62bb37
Oct 30 13:48:49.244954 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:49.244907    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-f4lw9-sg475" podStartSLOduration=1.583469628 podStartE2EDuration="4.244892072s" totalImagesPullingTime="2.661422444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:49.24360547 +0000 UTC m=+34.802607742" watchObservedRunningTime="2026-10-30 13:48:49.244892072 +0000 UTC m=+34.803894318"
Oct 30 13:48:49.283338 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:49.283262    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xpz2s" podStartSLOduration=2.049325585 podStartE2EDuration="5.283247224s" totalImagesPullingTime="3.233921639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:49.283177905 +0000 UTC m=+34.842180144" watchObservedRunningTime="2026-10-30 13:48:49.283247224 +0000 UTC m=+34.842249473"
Oct 30 13:48:50.196469 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:50.196417    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2w5dd" event={"ID":"f0bbec59-52f9-4dc6-b24b-169f666c08ae","Type":"ContainerStarted","Data":"b96ada9d2bf66950da4af8a81e33b90def208c4fcf20839770dda4172c62bb37"}
Oct 30 13:48:50.198589 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:50.198515    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t" event={"ID":"bbc411f7-86c3-4d69-8f20-af096604e14a","Type":"ContainerStarted","Data":"3fdaef211ad6e8793d46316f8f1fff250dc842d20ed55d740da4199df9261cc1"}
Oct 30 13:48:50.239123 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:50.239070    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t" podStartSLOduration=2.15040171 podStartE2EDuration="5.23905594s" totalImagesPullingTime="3.08865423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:50.238809352 +0000 UTC m=+35.797811601" watchObservedRunningTime="2026-10-30 13:48:50.23905594 +0000 UTC m=+35.798058191"
Oct 30 13:48:50.878717 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:50.878675    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sctxm_efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:51.060759 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:51.060725    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sctxm_efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:51.252116 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:51.252046    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sctxm_efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:51.640665 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:51.640637    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjg7z_62977d18-4b0e-4e1e-ba64-bcda9ca2095e/dns-node-resolver/0.log"
Oct 30 13:48:52.441216 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:52.441192    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6pbvr_08311d3a-d790-4761-96f0-e9a6b336e913/node-ca/0.log"
Oct 30 13:48:52.612516 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:52.612483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:52.615014 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:52.614982    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:52.671063 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:52.671033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:52.983403 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:52.983334    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nr98w"]
Oct 30 13:48:53.008891 ip-10-0-133-118 kubenswrapper[2580]: W1030 13:48:53.008863    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4819261_bdd9_419e_abd3_15fa171de861.slice/crio-05a8475a492015624c13631c7988cfb3572c46b81facc4608013d63672a615ad WatchSource:0}: Error finding container 05a8475a492015624c13631c7988cfb3572c46b81facc4608013d63672a615ad: Status 404 returned error can't find the container with id 05a8475a492015624c13631c7988cfb3572c46b81facc4608013d63672a615ad
Oct 30 13:48:53.115813 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:53.115784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:53.115922 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:53.115855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:53.115980 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:53.115945    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:53.116029 ip-10-0-133-118 kubenswrapper[2580]: E1030 13:48:53.116008    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:01.115987882 +0000 UTC m=+46.674990120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:53.116472 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:53.116444    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:53.213220 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:53.212780    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-98cgg" event={"ID":"fafdd535-5a88-4207-b2a4-cc43bf43bd17","Type":"ContainerStarted","Data":"457c272734bf9ce4842930b7f082d434c71b9ec4d1cce7d662945a2ab656b2bf"}
Oct 30 13:48:53.213220 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:53.212906    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:53.214268 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:53.214242    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nr98w" event={"ID":"b4819261-bdd9-419e-abd3-15fa171de861","Type":"ContainerStarted","Data":"05a8475a492015624c13631c7988cfb3572c46b81facc4608013d63672a615ad"}
Oct 30 13:48:53.251662 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:53.251578    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-98cgg" podStartSLOduration=34.383283901 podStartE2EDuration="38.251561602s" totalImagesPullingTime="3.868277701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:53.251480622 +0000 UTC m=+38.810482871" watchObservedRunningTime="2026-10-30 13:48:53.251561602 +0000 UTC m=+38.810563844"
Oct 30 13:48:53.441129 ip-10-0-133-118 kubenswrapper[2580]: I1030 13:48:53.441098    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xpz2s_6bd2311e-929b-4d5a-9857-47c0500c3be1/serve-healthcheck-canary/0.log"
Oct 30 13:48:54.908566 ip-10-0-133-118 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 13:48:54.927961 ip-10-0-133-118 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 13:48:54.928126 ip-10-0-133-118 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 13:48:54.928178 ip-10-0-133-118 systemd[1]: kubelet.service: Consumed 2.247s CPU time.
Oct 30 13:48:54.939267 ip-10-0-133-118 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 13:48:55.024291 ip-10-0-133-118 kubenswrapper[4530]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:48:55.024291 ip-10-0-133-118 kubenswrapper[4530]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 13:48:55.024291 ip-10-0-133-118 kubenswrapper[4530]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:48:55.024291 ip-10-0-133-118 kubenswrapper[4530]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:48:55.026396 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026314    4530 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:55.026396 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026396    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:55.026396 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026400    4530 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026403    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026406    4530 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026410    4530 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026414    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026417    4530 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026424    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026427    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026430    4530 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026432    4530 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026435    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026437    4530 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026442    4530 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026446    4530 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026449    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026452    4530 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026454    4530 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026457    4530 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026460    4530 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:55.026490 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026462    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026465    4530 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026468    4530 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026471    4530 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026473    4530 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026476    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026478    4530 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026481    4530 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026483    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026487    4530 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026489    4530 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026492    4530 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026496    4530 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026499    4530 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026502    4530 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026504    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026507    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026510    4530 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026513    4530 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026516    4530 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:55.026959 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026518    4530 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026521    4530 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026524    4530 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026527    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026529    4530 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026532    4530 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026534    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026537    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026556    4530 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026560    4530 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026566    4530 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026569    4530 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026572    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026574    4530 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026577    4530 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026580    4530 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026583    4530 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026586    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026588    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026591    4530 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:55.027438 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026594    4530 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026596    4530 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026599    4530 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026602    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026604    4530 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026607    4530 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026609    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026612    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026614    4530 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026617    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026620    4530 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026623    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026625    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026628    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026630    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026633    4530 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026635    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026638    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026641    4530 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:55.027923 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026643    4530 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026646    4530 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026649    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026651    4530 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026654    4530 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026657    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026660    4530 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026663    4530 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026666    4530 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026668    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026671    4530 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026674    4530 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026676    4530 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026678    4530 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026681    4530 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026683    4530 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026686    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026688    4530 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026691    4530 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026694    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:55.028406 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026696    4530 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026699    4530 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026701    4530 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026705    4530 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026707    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.026710    4530 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027496    4530 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027503    4530 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027507    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027510    4530 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027513    4530 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027516    4530 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027519    4530 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027521    4530 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027524    4530 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027527    4530 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027530    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027533    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027536    4530 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:55.028933 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027554    4530 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027557    4530 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027559    4530 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027562    4530 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027565    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027568    4530 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027570    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027573    4530 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027576    4530 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027579    4530 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027581    4530 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027584    4530 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027586    4530 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027589    4530 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027591    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027594    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027597    4530 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027599    4530 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027601    4530 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027610    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:55.029387 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027613    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027616    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027618    4530 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027621    4530 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027624    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027626    4530 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027629    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027632    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027634    4530 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027637    4530 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027640    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027642    4530 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027645    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027647    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027650    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027652    4530 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027655    4530 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027657    4530 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027660    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:55.029894 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027662    4530 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027665    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027669    4530 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027673    4530 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027676    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027679    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027682    4530 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027684    4530 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027687    4530 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027689    4530 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027692    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027695    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027698    4530 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027706    4530 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027709    4530 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027712    4530 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027714    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027717    4530 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027720    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:55.030689 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027722    4530 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027725    4530 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027727    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027730    4530 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027733    4530 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027736    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027739    4530 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027741    4530 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027745    4530 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027748    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027751    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027754    4530 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027756    4530 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027759    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027761    4530 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027764    4530 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027766    4530 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027773    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027776    4530 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:55.031358 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027779    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027782    4530 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027784    4530 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027787    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027789    4530 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027792    4530 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027794    4530 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027797    4530 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027805    4530 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027808    4530 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027810    4530 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027813    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027816    4530 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027818    4530 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027821    4530 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.027823    4530 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028376    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028383    4530 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028387    4530 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028390    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:55.031850 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028394    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028396    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028400    4530 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028403    4530 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028405    4530 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028408    4530 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028411    4530 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028414    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028416    4530 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028420    4530 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028423    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028425    4530 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028428    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028431    4530 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028433    4530 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028436    4530 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028438    4530 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028441    4530 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028444    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028446    4530 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:55.032316 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028455    4530 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028458    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028460    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028470    4530 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028473    4530 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028476    4530 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028478    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028481    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028483    4530 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028486    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028488    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028491    4530 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028493    4530 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028498    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028501    4530 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028503    4530 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028507    4530 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028509    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028512    4530 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:55.032815 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028515    4530 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028517    4530 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028522    4530 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028525    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028527    4530 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028530    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028533    4530 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028536    4530 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028551    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028554    4530 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028557    4530 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028560    4530 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028562    4530 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028565    4530 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028573    4530 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028576    4530 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028579    4530 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028581    4530 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028584    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028586    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:55.033277 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028589    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028592    4530 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028594    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028597    4530 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028599    4530 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028602    4530 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028606    4530 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028608    4530 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028611    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028613    4530 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028616    4530 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028619    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028621    4530 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028624    4530 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028628    4530 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028630    4530 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028633    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028636    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028638    4530 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028641    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:55.033810 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028643    4530 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028646    4530 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028648    4530 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028652    4530 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028656    4530 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028658    4530 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028661    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028681    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028686    4530 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028690    4530 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028693    4530 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028698    4530 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028705    4530 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028711    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028715    4530 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028718    4530 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028720    4530 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028723    4530 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028727    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:55.034378 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028730    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028732    4530 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028735    4530 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.028737    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028851    4530 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028861    4530 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028871    4530 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028879    4530 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028884    4530 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028888    4530 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028892    4530 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028896    4530 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028899    4530 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028902    4530 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028906    4530 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028909    4530 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028912    4530 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028915    4530 flags.go:64] FLAG: --cgroup-root=""
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028918    4530 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028921    4530 flags.go:64] FLAG: --client-ca-file=""
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028924    4530 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028933    4530 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028939    4530 flags.go:64] FLAG: --cluster-domain=""
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028942    4530 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 13:48:55.035229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028945    4530 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028948    4530 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028952    4530 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028955    4530 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028958    4530 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028962    4530 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028965    4530 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028969    4530 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028972    4530 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028975    4530 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028978    4530 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028981    4530 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028986    4530 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028989    4530 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028992    4530 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028996    4530 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.028999    4530 flags.go:64] FLAG: --enable-server="true"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029002    4530 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029006    4530 flags.go:64] FLAG: --event-burst="100"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029010    4530 flags.go:64] FLAG: --event-qps="50"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029013    4530 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029016    4530 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029019    4530 flags.go:64] FLAG: --eviction-hard=""
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029023    4530 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 13:48:55.036274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029026    4530 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029029    4530 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029032    4530 flags.go:64] FLAG: --eviction-soft=""
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029036    4530 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029039    4530 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029041    4530 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029045    4530 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029048    4530 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029051    4530 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029054    4530 flags.go:64] FLAG: --feature-gates=""
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029060    4530 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029063    4530 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029066    4530 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029070    4530 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029073    4530 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029077    4530 flags.go:64] FLAG: --help="false"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029080    4530 flags.go:64] FLAG: --hostname-override="ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029083    4530 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029086    4530 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029090    4530 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029093    4530 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029096    4530 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029099    4530 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 13:48:55.037432 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029103    4530 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029106    4530 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029109    4530 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029112    4530 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029115    4530 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029118    4530 flags.go:64] FLAG: --kube-reserved=""
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029121    4530 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029124    4530 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029127    4530 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029130    4530 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029133    4530 flags.go:64] FLAG: --lock-file=""
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029136    4530 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029139    4530 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029142    4530 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029147    4530 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029150    4530 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029154    4530 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029157    4530 flags.go:64] FLAG: --logging-format="text"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029159    4530 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029163    4530 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029166    4530 flags.go:64] FLAG: --manifest-url=""
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029169    4530 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029173    4530 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029176    4530 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029181    4530 flags.go:64] FLAG: --max-pods="110"
Oct 30 13:48:55.038413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029185    4530 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029188    4530 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029191    4530 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029194    4530 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029197    4530 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029200    4530 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029203    4530 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029211    4530 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029214    4530 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029217    4530 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029220    4530 flags.go:64] FLAG: --pod-cidr=""
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029223    4530 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029227    4530 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029230    4530 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029233    4530 flags.go:64] FLAG: --port="10250"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029236    4530 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029239    4530 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ec13e27d2b23feb0"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029242    4530 flags.go:64] FLAG: --qos-reserved=""
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029245    4530 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029248    4530 flags.go:64] FLAG: --register-node="true"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029251    4530 flags.go:64] FLAG: --register-with-taints=""
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029255    4530 flags.go:64] FLAG: --registry-burst="10"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029258    4530 flags.go:64] FLAG: --registry-qps="5"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029261    4530 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029264    4530 flags.go:64] FLAG: --reserved-memory=""
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029268    4530 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 13:48:55.039453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029271    4530 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029274    4530 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029277    4530 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029280    4530 flags.go:64] FLAG: --runonce="false"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029283    4530 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029288    4530 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029291    4530 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029294    4530 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029297    4530 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029300    4530 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029303    4530 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029306    4530 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029309    4530 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029313    4530 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029316    4530 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029319    4530 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029322    4530 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029325    4530 flags.go:64] FLAG: --system-cgroups=""
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029328    4530 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029333    4530 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029336    4530 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029339    4530 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029343    4530 flags.go:64] FLAG: --tls-min-version=""
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029346    4530 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029349    4530 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 13:48:55.040621 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029352    4530 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029358    4530 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029361    4530 flags.go:64] FLAG: --v="2"
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029365    4530 flags.go:64] FLAG: --version="false"
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029369    4530 flags.go:64] FLAG: --vmodule=""
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029374    4530 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029377    4530 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029522    4530 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029527    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029530    4530 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029533    4530 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029535    4530 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029553    4530 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029558    4530 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029562    4530 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029566    4530 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029568    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029571    4530 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029574    4530 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029577    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029580    4530 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:55.041706 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029583    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029586    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029588    4530 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029591    4530 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029593    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029596    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029599    4530 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029601    4530 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029604    4530 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029607    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029609    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029612    4530 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029615    4530 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029617    4530 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029620    4530 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029622    4530 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029625    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029627    4530 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029631    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029633    4530 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:55.042642 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029636    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029638    4530 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029641    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029646    4530 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029649    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029653    4530 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029655    4530 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029658    4530 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029661    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029664    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029666    4530 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029670    4530 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029673    4530 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029675    4530 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029678    4530 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029680    4530 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029683    4530 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029685    4530 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029688    4530 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:55.043201 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029691    4530 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029694    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029696    4530 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029699    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029701    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029704    4530 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029706    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029709    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029711    4530 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029713    4530 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029716    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029719    4530 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029722    4530 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029725    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029728    4530 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029730    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029734    4530 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029737    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029740    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029742    4530 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:55.043889 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029744    4530 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029747    4530 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029750    4530 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029753    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029757    4530 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029759    4530 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029762    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029764    4530 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029767    4530 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029770    4530 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029772    4530 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029775    4530 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029777    4530 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029780    4530 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029782    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029786    4530 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029788    4530 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029791    4530 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029794    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029796    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:55.044711 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029799    4530 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029802    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029804    4530 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029807    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029809    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029812    4530 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029815    4530 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029817    4530 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029822    4530 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029826    4530 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029829    4530 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029833    4530 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.029836    4530 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.029862    4530 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030056    4530 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030087    4530 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:55.045279 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030093    4530 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030098    4530 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030102    4530 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030108    4530 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030112    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030116    4530 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030121    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030126    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030130    4530 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030134    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030144    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030148    4530 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030153    4530 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030157    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030162    4530 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030166    4530 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030170    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030174    4530 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030179    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:55.045919 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030183    4530 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030188    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030192    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030205    4530 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030214    4530 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030219    4530 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030224    4530 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030228    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030232    4530 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030236    4530 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030241    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030245    4530 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030250    4530 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030254    4530 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030259    4530 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030263    4530 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030267    4530 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030276    4530 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030280    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030286    4530 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:55.046527 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030290    4530 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030294    4530 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030298    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030302    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030307    4530 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030314    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030319    4530 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030324    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030334    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030338    4530 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030342    4530 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030346    4530 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030352    4530 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030358    4530 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030364    4530 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030368    4530 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030373    4530 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030378    4530 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030383    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:55.047104 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030398    4530 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030415    4530 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030421    4530 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030425    4530 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030429    4530 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030433    4530 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030438    4530 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030442    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030447    4530 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030451    4530 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030455    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030459    4530 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030464    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030468    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030476    4530 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030481    4530 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030485    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030489    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030494    4530 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030498    4530 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:55.047558 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030503    4530 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030507    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030511    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030515    4530 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030520    4530 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030524    4530 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030529    4530 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030551    4530 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030556    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030560    4530 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030565    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030570    4530 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030574    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030578    4530 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030582    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030592    4530 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030598    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030603    4530 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030608    4530 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:55.048047 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030617    4530 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030622    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030626    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030630    4530 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030634    4530 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030638    4530 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.030642    4530 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.034401    4530 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.034416    4530 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.034434    4530 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.034438    4530 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034500    4530 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034506    4530 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034510    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034513    4530 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034516    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034519    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034522    4530 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:55.048485 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034525    4530 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034528    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034530    4530 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034533    4530 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034535    4530 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034552    4530 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034555    4530 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034558    4530 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034560    4530 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034563    4530 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034566    4530 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034569    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034572    4530 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034574    4530 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034577    4530 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034579    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034582    4530 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034585    4530 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034590    4530 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:55.048981 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034594    4530 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034598    4530 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034601    4530 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034604    4530 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034607    4530 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034610    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034613    4530 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034615    4530 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034618    4530 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034621    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034623    4530 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034626    4530 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034629    4530 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034631    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034634    4530 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034636    4530 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034639    4530 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034642    4530 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034645    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:55.049432 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034647    4530 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034650    4530 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034653    4530 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034655    4530 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034658    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034661    4530 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034663    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034666    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034668    4530 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034671    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034674    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034677    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034680    4530 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034683    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034686    4530 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034688    4530 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034691    4530 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034694    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034696    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034699    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:55.049898 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034702    4530 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034704    4530 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034707    4530 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034709    4530 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034712    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034714    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034718    4530 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034720    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034723    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034726    4530 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034728    4530 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034731    4530 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034733    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034736    4530 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034740    4530 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034745    4530 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034748    4530 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034751    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034754    4530 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034757    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:55.050360 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034759    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034763    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034765    4530 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034769    4530 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034771    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034774    4530 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034777    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034779    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034782    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034784    4530 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034787    4530 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034790    4530 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034793    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034795    4530 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034798    4530 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034800    4530 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034803    4530 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034806    4530 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034808    4530 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034812    4530 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:55.050899 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034815    4530 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.034842    4530 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034975    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034979    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034983    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034986    4530 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034989    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034992    4530 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034995    4530 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.034999    4530 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035003    4530 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035006    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035009    4530 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035012    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035015    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:55.051855 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035019    4530 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035021    4530 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035024    4530 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035027    4530 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035030    4530 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035033    4530 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035035    4530 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035038    4530 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035041    4530 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035044    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035046    4530 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035049    4530 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035051    4530 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035054    4530 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035056    4530 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035059    4530 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035061    4530 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035064    4530 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035067    4530 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:55.052423 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035069    4530 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035072    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035074    4530 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035077    4530 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035080    4530 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035083    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035085    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035088    4530 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035090    4530 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035093    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035095    4530 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035098    4530 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035100    4530 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035103    4530 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035106    4530 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035108    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035111    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035113    4530 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035116    4530 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035118    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:55.053220 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035121    4530 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035123    4530 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035126    4530 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035128    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035131    4530 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035133    4530 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035136    4530 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035139    4530 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035141    4530 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035144    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035146    4530 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035149    4530 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035152    4530 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035154    4530 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035157    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035160    4530 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035162    4530 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035165    4530 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035168    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035171    4530 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:55.053764 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035175    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035177    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035180    4530 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035182    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035185    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035188    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035190    4530 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035195    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035198    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035201    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035203    4530 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035206    4530 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035208    4530 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035211    4530 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035213    4530 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035216    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035218    4530 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035221    4530 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035223    4530 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035226    4530 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:55.054260 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035228    4530 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035231    4530 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035233    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035236    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035239    4530 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035242    4530 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035245    4530 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035248    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035251    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035253    4530 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035256    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035258    4530 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035261    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035263    4530 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.035286    4530 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:48:55.054776 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035316    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035321    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035323    4530 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035327    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035331    4530 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035334    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035336    4530 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035339    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035341    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035344    4530 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035347    4530 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035349    4530 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035354    4530 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035357    4530 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035361    4530 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035365    4530 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035368    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035371    4530 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035373    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:48:55.055139 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035376    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035378    4530 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035381    4530 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035385    4530 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035387    4530 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035390    4530 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035392    4530 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035395    4530 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035398    4530 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035400    4530 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035403    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035405    4530 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035408    4530 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035411    4530 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035413    4530 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035416    4530 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035418    4530 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035422    4530 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035425    4530 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:48:55.055626 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035427    4530 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035430    4530 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035433    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035435    4530 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035438    4530 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035440    4530 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035444    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035447    4530 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035450    4530 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035452    4530 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035455    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035457    4530 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035460    4530 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035462    4530 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035465    4530 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035468    4530 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035471    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035474    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035476    4530 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035479    4530 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:48:55.056107 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035482    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035484    4530 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035487    4530 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035489    4530 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035492    4530 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035494    4530 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035498    4530 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035500    4530 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035503    4530 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035505    4530 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035509    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035512    4530 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035515    4530 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035517    4530 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035520    4530 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035522    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035525    4530 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035528    4530 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035531    4530 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035534    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:48:55.056581 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035559    4530 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035566    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035570    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035572    4530 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035575    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035578    4530 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035581    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035584    4530 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035586    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035590    4530 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035593    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035596    4530 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035599    4530 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035601    4530 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035604    4530 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035606    4530 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035609    4530 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035611    4530 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035614    4530 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035616    4530 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:48:55.057060 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035619    4530 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035621    4530 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035625    4530 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035628    4530 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035630    4530 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035633    4530 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035636    4530 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:55.035638    4530 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.035687    4530 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.036629    4530 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.036675    4530 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.036852    4530 server.go:1014] "Starting client certificate rotation"
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.037006    4530 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.037062    4530 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 13:43:15 +0000 UTC" deadline="2028-04-01 12:00:58.81074407 +0000 UTC"
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.037096    4530 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12454h12m3.773650325s"
Oct 30 13:48:55.057601 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.037285    4530 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:48:55.057970 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.038373    4530 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:48:55.057970 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.039662    4530 log.go:25] "Validated CRI v1 runtime API"
Oct 30 13:48:55.057970 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.041154    4530 log.go:25] "Validated CRI v1 image API"
Oct 30 13:48:55.057970 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.041935    4530 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 13:48:55.057970 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.043354    4530 fs.go:135] Filesystem UUIDs: map[1b383948-1a3b-4c69-85e9-a21c409e3e12:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f488d9b9-b66f-4f13-8cdf-ac9e22d4ec79:/dev/nvme0n1p3]
Oct 30 13:48:55.058106 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.043378    4530 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01204cb7fd2d72f4b3ac92a86bbb962182f4b6cb7e3dbc9d98c0e6dd9c37aaa3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01204cb7fd2d72f4b3ac92a86bbb962182f4b6cb7e3dbc9d98c0e6dd9c37aaa3/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05a8475a492015624c13631c7988cfb3572c46b81facc4608013d63672a615ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05a8475a492015624c13631c7988cfb3572c46b81facc4608013d63672a615ad/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dff1ceb0266e2bc94215f7a694fb1716ab5455670d27463ec082a8364403289/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dff1ceb0266e2bc94215f7a694fb1716ab5455670d27463ec082a8364403289/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fc3828e972ad5a88dd2594e0093f91696fc979c397cba6d0a48747b9a081914/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fc3828e972ad5a88dd2594e0093f91696fc979c397cba6d0a48747b9a081914/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b8733a18ac8fac79d29356d289d071b29322a192622aa19a50cfd10b43522b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b8733a18ac8fac79d29356d289d071b29322a192622aa19a50cfd10b43522b7/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/677a40a55c1c06e74ac7bcd6e73bd1403940c55684f46ee92d1754353be246a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/677a40a55c1c06e74ac7bcd6e73bd1403940c55684f46ee92d1754353be246a3/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7031183246a934e66f3178646e691d7ad66cdb00d8b9eb727fadfd003e8e90e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7031183246a934e66f3178646e691d7ad66cdb00d8b9eb727fadfd003e8e90e3/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74d518a74273128a954146e126cbc81fcdf372a21e04a3ee7a777d3e8840a907/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74d518a74273128a954146e126cbc81fcdf372a21e04a3ee7a777d3e8840a907/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e7b183942858b307e989212f04f7088a104787ee77e63ff40852eb888bc702d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e7b183942858b307e989212f04f7088a104787ee77e63ff40852eb888bc702d/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9364e815e66d572732f722076f92640b815f70566b1130b7142f9eaef15437c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9364e815e66d572732f722076f92640b815f70566b1130b7142f9eaef15437c4/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ccc6aaf2a1f82e3bbdff57aaaa3356058de2312fd2e2a4d0f482734d0526a05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ccc6aaf2a1f82e3bbdff57aaaa3356058de2312fd2e2a4d0f482734d0526a05/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b96ada9d2bf66950da4af8a81e33b90def208c4fcf20839770dda4172c62bb37/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b96ada9d2bf66950da4af8a81e33b90def208c4fcf20839770dda4172c62bb37/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5c2a0f95a9017cab248b90d1c1dd73282463406477893e93d7dd2177e9bcc24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5c2a0f95a9017cab248b90d1c1dd73282463406477893e93d7dd2177e9bcc24/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c71db214edb721aea8aa75d032c94def95d0604d9ae341a7dc3e163700c4777a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c71db214edb721aea8aa75d032c94def95d0604d9ae341a7dc3e163700c4777a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd5bf80e3d7cf6dabf691d654b4a105f0793c63e825814348f28e3c752dc9222/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd5bf80e3d7cf6dabf691d654b4a105f0793c63e825814348f28e3c752dc9222/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e55e12210c8cd3e333f4f252c97f03006b1cfb7788fc7250a14725f39ef44e2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e55e12210c8cd3e333f4f252c97f03006b1cfb7788fc7250a14725f39ef44e2d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/055c8a12-fcb0-440d-bacc-20dd67b2ce02/volumes/kubernetes.io~projected/kube-api-access-6k6s7:{mountpoint:/var/lib/kubelet/pods/055c8a12-fcb0-440d-bacc-20dd67b2ce02/volumes/kubernetes.io~projected/kube-api-access-6k6s7 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08311d3a-d790-4761-96f0-e9a6b336e913/volumes/kubernetes.io~projected/kube-api-access-9f6hc:{mountpoint:/var/lib/kubelet/pods/08311d3a-d790-4761-96f0-e9a6b336e913/volumes/kubernetes.io~projected/kube-api-access-9f6hc major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b3dcf54-ba8f-4080-9666-cb41a151ccad/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0b3dcf54-ba8f-4080-9666-cb41a151ccad/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b3dcf54-ba8f-4080-9666-cb41a151ccad/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0b3dcf54-ba8f-4080-9666-cb41a151ccad/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b3dcf54-ba8f-4080-9666-cb41a151ccad/volumes/kubernetes.io~projected/kube-api-access-rp28b:{mountpoint:/var/lib/kubelet/pods/0b3dcf54-ba8f-4080-9666-cb41a151ccad/volumes/kubernetes.io~projected/kube-api-access-rp28b major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c/volumes/kubernetes.io~projected/kube-api-access-2rnlm:{mountpoint:/var/lib/kubelet/pods/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c/volumes/kubernetes.io~projected/kube-api-access-2rnlm major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/375bd1fd-307b-4a6b-b6be-99b492065982/volumes/kubernetes.io~projected/kube-api-access-tdwg4:{mountpoint:/var/lib/kubelet/pods/375bd1fd-307b-4a6b-b6be-99b492065982/volumes/kubernetes.io~projected/kube-api-access-tdwg4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51c2e87b-ea89-48dd-9dd0-d6c674e0f837/volumes/kubernetes.io~projected/kube-api-access-2tz4h:{mountpoint:/var/lib/kubelet/pods/51c2e87b-ea89-48dd-9dd0-d6c674e0f837/volumes/kubernetes.io~projected/kube-api-access-2tz4h major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62977d18-4b0e-4e1e-ba64-bcda9ca2095e/volumes/kubernetes.io~projected/kube-api-access-kfk95:{mountpoint:/var/lib/kubelet/pods/62977d18-4b0e-4e1e-ba64-bcda9ca2095e/volumes/kubernetes.io~projected/kube-api-access-kfk95 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bd2311e-929b-4d5a-9857-47c0500c3be1/volumes/kubernetes.io~projected/kube-api-access-qlxg8:{mountpoint:/var/lib/kubelet/pods/6bd2311e-929b-4d5a-9857-47c0500c3be1/volumes/kubernetes.io~projected/kube-api-access-qlxg8 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bd2311e-929b-4d5a-9857-47c0500c3be1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6bd2311e-929b-4d5a-9857-47c0500c3be1/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9acb4199-6d58-4d3d-86dc-96326517baf8/volumes/kubernetes.io~projected/kube-api-access-lz5fm:{mountpoint:/var/lib/kubelet/pods/9acb4199-6d58-4d3d-86dc-96326517baf8/volumes/kubernetes.io~projected/kube-api-access-lz5fm major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b37f6a9a-e81e-40a1-afd4-2726e0bf2614/volumes/kubernetes.io~projected/kube-api-access-xpxql:{mountpoint:/var/lib/kubelet/pods/b37f6a9a-e81e-40a1-afd4-2726e0bf2614/volumes/kubernetes.io~projected/kube-api-access-xpxql major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4819261-bdd9-419e-abd3-15fa171de861/volumes/kubernetes.io~projected/kube-api-access-dkn6d:{mountpoint:/var/lib/kubelet/pods/b4819261-bdd9-419e-abd3-15fa171de861/volumes/kubernetes.io~projected/kube-api-access-dkn6d major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4819261-bdd9-419e-abd3-15fa171de861/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/b4819261-bdd9-419e-abd3-15fa171de861/volumes/kubernetes.io~secret/metrics-tls major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbc411f7-86c3-4d69-8f20-af096604e14a/volumes/kubernetes.io~projected/kube-api-access-5vcdc:{mountpoint:/var/lib/kubelet/pods/bbc411f7-86c3-4d69-8f20-af096604e14a/volumes/kubernetes.io~projected/kube-api-access-5vcdc major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbc411f7-86c3-4d69-8f20-af096604e14a/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/bbc411f7-86c3-4d69-8f20-af096604e14a/volumes/kubernetes.io~secret/signing-key major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6f44206-dd53-4dae-b68c-84a8317227b5/volumes/kubernetes.io~projected/kube-api-access-swft4:{mountpoint:/var/lib/kubelet/pods/c6f44206-dd53-4dae-b68c-84a8317227b5/volumes/kubernetes.io~projected/kube-api-access-swft4 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7926b85-83e3-4e81-a403-774ea8daebe1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d7926b85-83e3-4e81-a403-774ea8daebe1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7926b85-83e3-4e81-a403-774ea8daebe1/volumes/kubernetes.io~projected/kube-api-access-8zvkx:{mountpoint:/var/lib/kubelet/pods/d7926b85-83e3-4e81-a403-774ea8daebe1/volumes/kubernetes.io~projected/kube-api-access-8zvkx major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7926b85-83e3-4e81-a403-774ea8daebe1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d7926b85-83e3-4e81-a403-774ea8daebe1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0bbec59-52f9-4dc6-b24b-169f666c08ae/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f0bbec59-52f9-4dc6-b24b-169f666c08ae/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fafdd535-5a88-4207-b2a4-cc43bf43bd17/volumes/kubernetes.io~projected/kube-api-access-44fn2:{mountpoint:/var/lib/kubelet/pods/fafdd535-5a88-4207-b2a4-cc43bf43bd17/volumes/kubernetes.io~projected/kube-api-access-44fn2 major:0 minor:233 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a6d2098fa5774e197ad1c79861fccecc627447330da7640f3954d28ca1d35c4b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/275baf0e3edf180b9f9965de4319b42ad881481f0409e5f18fbf6c67ed0ba445/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4dd3abacd07d4406f0bd0ab968a631768a295fadf85d384117e9c4cd043ce3fb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ae1821c34b178f4e49a195e17c11e47a7a4a84e2fbb8bfc94ce58b2b5231ac44/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/092ea769955b83806da0f2c0e9f9baef8c1f03e19a5987479952da119260543f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/bbd297d505fe98828fbb04e9993ed335781c85419c72416ecdb6974f9c47b699/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/af965a5eba30faea1e8b161470ed0b86e2c1fcb23a074bc77f64e029696fd68e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1449c64edef480b4a89183107d600f614d7b0126928072d6f0badede5eea5994/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/9e00d2e8f8c5e8a07f410656fddb4d4aa156f8ef6590f27c6f75a6fa5522b826/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/deba88377ce11407c749ee838dacb02733126fd0745b06f44e3342651cf0f8db/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f349c93f88d6f9e56e5868a473a207ef69d95b178bfb64d514279286aa033512/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/2169a75473f7cd3dae0f18ba1c7bd1eda8cb8bb713481405defad76c3c80f24e/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d8a719b76fcd58eeb167a46e01e370a5aa7dc5a189d8f24e9cbc4ea74d88f9b1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f3426b9b68dc33ed12d7d43c2382dec4ab5365228839dd58d30087b58ac4884e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d060fade964e7bcca65a82c83bc8245efbcf185ab9325932926307edf084a04f/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c701175655567111cb171e4cdfb2a455b328ce0a4df8fb5c5b62db454be6da92/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/90753cb68aae68b9655a8ae8cff004bf8ee190ed03a2cba187c5be8c85dfa468/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4861449390dbd0a3c4c3c9d0dd5fc5fb44375b2ce7691146bcbf603758cd0a7e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/39ecf64d7fdfc4affaf56a59e49a80e143a4fbaa03780b358a0ae547fca35e11/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f87dd8abd4dd55af63f7138ec240758c423c1340d1dfa8709f8ad496f7b62533/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/bce7aaf6fec0a682bdfdb0474c7cfca66e1aeb1605668c8e8a3d73fab4ec30ff/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/f475bbb713bbd1c70feeed55497a4f29791506c911cd438bbbdcb2aae3fd0827/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/0b0c345d8722c3fd7bde48a92c04771c523cf8085d4e1599793a7ecac10d04d1/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/81773129fc547989b07db2a107a07c680cf0b299291e2bcc71cc1865916cc133/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/4a6be800c387356841528cc3cbb456094b844c3c43e69f95d319b3d25998cb92/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/17301b2bad5ec645c7deaa6d0035a5ecfbfb0f4c8bac282feaa391f8299869a3/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/8bd8e187b35de7955094a947333c406defb2c04ca1792f9bc15a6e202f2befff/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/8f4764c3ed4ca79a9aa237c242bd7e02e4beae4e6d16a807e93f3cce0a0edcf7/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/dfaae7aea111cd5d247c62985ec65691855a01972f393ca39ee36b277f555322/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/4ec1159c04f177f6f5036c24e6e9f787567eb46762c2986ef322dfbddba04af9/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/fec36f0e44a6f4e8e13430dd503b1f6008d102254e5a6b119db3d6a3b0e56347/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/ad8fe0b3bbce52b6925340a87d0ce97ecd203240428b761b1e47b74d44b05477/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/228e44107ce9395569686cfcaef8278faeb48d95ac60e8ac515920ac38aa1c10/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c93b05d11458b2502ccd235eff1408683606d4281749daf1e064c60bd81e98d5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5e2eab287ebbcfec348769087c8d9bd88ac5d5b97494077f084165fb1cfbb535/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/77479ace884a04894d1cf25abb2786e82673e092b74db688ac727520e561b9ba/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/be45455f4fccb0fcc478b7e7eda4c905d6f5f21f511b895cf223f5a3d71126b6/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/b9562415f0271d1b0667f50dfab1305da24d76fbf7d63c863c3635093b6bdfd1/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/579d003566ece8bfc7421c262d280e8b600f0006291a862dbb2e97dea5d3efbd/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/2b3920705bda6ba10b0176eae0530bc026993414994593f15c9fe3be9e12dd4f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/7bf8f8397eaac9a5cb94452a73678e4a674aa1ae173299a559b7c1bccbdf36e8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a6d70932a33faafd9abf7976dc87bebdcd7bde18d684ed68d36d38666eb7b3e5/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.050617    4530 manager.go:217] Machine: {Timestamp:2026-10-30 13:48:55.050152302 +0000 UTC m=+0.082051232 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099969 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d23b13ea07d5f567963215179f05c SystemUUID:ec2d23b1-3ea0-7d5f-5679-63215179f05c BootID:f3fb7500-0f47-446e-9dc2-5ff832c24c71 Filesystems:[{Device:/var/lib/kubelet/pods/6bd2311e-929b-4d5a-9857-47c0500c3be1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c71db214edb721aea8aa75d032c94def95d0604d9ae341a7dc3e163700c4777a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08311d3a-d790-4761-96f0-e9a6b336e913/volumes/kubernetes.io~projected/kube-api-access-9f6hc DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7926b85-83e3-4e81-a403-774ea8daebe1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/375bd1fd-307b-4a6b-b6be-99b492065982/volumes/kubernetes.io~projected/kube-api-access-tdwg4 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9acb4199-6d58-4d3d-86dc-96326517baf8/volumes/kubernetes.io~projected/kube-api-access-lz5fm DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e55e12210c8cd3e333f4f252c97f03006b1cfb7788fc7250a14725f39ef44e2d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4819261-bdd9-419e-abd3-15fa171de861/volumes/kubernetes.io~projected/kube-api-access-dkn6d DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7926b85-83e3-4e81-a403-774ea8daebe1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6f44206-dd53-4dae-b68c-84a8317227b5/volumes/kubernetes.io~projected/kube-api-access-swft4 DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05a8475a492015624c13631c7988cfb3572c46b81facc4608013d63672a615ad/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9364e815e66d572732f722076f92640b815f70566b1130b7142f9eaef15437c4/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/677a40a55c1c06e74ac7bcd6e73bd1403940c55684f46ee92d1754353be246a3/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dff1ceb0266e2bc94215f7a694fb1716ab5455670d27463ec082a8364403289/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b3dcf54-ba8f-4080-9666-cb41a151ccad/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e7b183942858b307e989212f04f7088a104787ee77e63ff40852eb888bc702d/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fafdd535-5a88-4207-b2a4-cc43bf43bd17/volumes/kubernetes.io~projected/kube-api-access-44fn2 DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b4819261-bdd9-419e-abd3-15fa171de861/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:242 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5c2a0f95a9017cab248b90d1c1dd73282463406477893e93d7dd2177e9bcc24/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0bbec59-52f9-4dc6-b24b-169f666c08ae/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b37f6a9a-e81e-40a1-afd4-2726e0bf2614/volumes/kubernetes.io~projected/kube-api-access-xpxql DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b3dcf54-ba8f-4080-9666-cb41a151ccad/volumes/kubernetes.io~projected/kube-api-access-rp28b DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/055c8a12-fcb0-440d-bacc-20dd67b2ce02/volumes/kubernetes.io~projected/kube-api-access-6k6s7 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b96ada9d2bf66950da4af8a81e33b90def208c4fcf20839770dda4172c62bb37/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fc3828e972ad5a88dd2594e0093f91696fc979c397cba6d0a48747b9a081914/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bd2311e-929b-4d5a-9857-47c0500c3be1/volumes/kubernetes.io~projected/kube-api-access-qlxg8 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bbc411f7-86c3-4d69-8f20-af096604e14a/volumes/kubernetes.io~projected/kube-api-access-5vcdc DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7926b85-83e3-4e81-a403-774ea8daebe1/volumes/kubernetes.io~projected/kube-api-access-8zvkx DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74d518a74273128a954146e126cbc81fcdf372a21e04a3ee7a777d3e8840a907/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7031183246a934e66f3178646e691d7ad66cdb00d8b9eb727fadfd003e8e90e3/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ccc6aaf2a1f82e3bbdff57aaaa3356058de2312fd2e2a4d0f482734d0526a05/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0b3dcf54-ba8f-4080-9666-cb41a151ccad/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b8733a18ac8fac79d29356d289d071b29322a192622aa19a50cfd10b43522b7/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51c2e87b-ea89-48dd-9dd0-d6c674e0f837/volumes/kubernetes.io~projected/kube-api-access-2tz4h DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbc411f7-86c3-4d69-8f20-af096604e14a/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd5bf80e3d7cf6dabf691d654b4a105f0793c63e825814348f28e3c752dc9222/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c/volumes/kubernetes.io~projected/kube-api-access-2rnlm DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/62977d18-4b0e-4e1e-ba64-bcda9ca2095e/volumes/kubernetes.io~projected/kube-api-access-kfk95 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01204cb7fd2d72f4b3ac92a86bbb962182f4b6cb7e3dbc9d98c0e6dd9c37aaa3/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:05a8475a4920156 MacAddress:ee:c9:fc:33:e7:f2 Speed:10000 Mtu:8901} {Name:7031183246a934e MacAddress:66:a3:e7:fa:bd:30 Speed:10000 Mtu:8901} {Name:74d518a74273128 MacAddress:5e:73:99:e3:5b:ef Speed:10000 Mtu:8901} {Name:7e7b183942858b3 MacAddress:9e:4c:ec:8a:c7:ca Speed:10000 Mtu:8901} {Name:b96ada9d2bf6695 MacAddress:ea:3e:97:1e:cc:38 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e3:7a:80:b5 Speed:0 Mtu:9001} {Name:br-int MacAddress:82:98:84:dd:e9:26 Speed:0 Mtu:8901} {Name:dd5bf80e3d7cf6d MacAddress:ce:15:5e:ea:8e:f2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e3:7a:80:b5 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:56:b2:dc:4d:ef:b2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:de:5f:80:91:a3:cc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.050866    4530 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.050908    4530 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.051249    4530 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.051288    4530 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-118.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.051468    4530 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.051481    4530 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.051504    4530 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.051531    4530 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.051626    4530 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.051851    4530 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.051981    4530 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.052024    4530 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.052043    4530 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.052059    4530 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.052071    4530 kubelet.go:405] "Adding apiserver pod source"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.052081    4530 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.052836    4530 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.052849    4530 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 13:48:55.058245 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.053374    4530 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 13:48:55.062520 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062505    4530 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 13:48:55.062520 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062520    4530 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 13:48:55.062634 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062624    4530 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 13:48:55.062805 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062794    4530 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 13:48:55.062836 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062823    4530 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 13:48:55.062912 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062904    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 13:48:55.062941 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062918    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 13:48:55.062941 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062924    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 13:48:55.062941 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062929    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 13:48:55.062941 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062934    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 13:48:55.062941 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062940    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 13:48:55.063094 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062945    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 13:48:55.063094 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062951    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 13:48:55.063094 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062958    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 13:48:55.063094 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062963    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 13:48:55.063094 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062981    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 13:48:55.063094 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.062990    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 13:48:55.063094 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.063015    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 13:48:55.063094 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.063020    4530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 13:48:55.066165 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.066153    4530 server.go:1263] "Started kubelet"
Oct 30 13:48:55.067988 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.066242    4530 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 13:48:55.067988 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.066275    4530 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 13:48:55.067988 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.066485    4530 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 13:48:55.067988 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.067849    4530 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 13:48:55.066688 ip-10-0-133-118 systemd[1]: Started Kubernetes Kubelet.
Oct 30 13:48:55.068356 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.068329    4530 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 13:48:55.071570 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.071209    4530 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 13:48:55.071570 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.071295    4530 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 13:48:55.071570 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.071492    4530 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 13:48:55.071570 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.071481    4530 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 13:43:17 +0000 UTC" deadline="2028-05-26 13:05:56.319492644 +0000 UTC"
Oct 30 13:48:55.071570 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.071512    4530 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13775h17m1.247983882s"
Oct 30 13:48:55.071570 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:55.071492    4530 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found"
Oct 30 13:48:55.071913 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.071651    4530 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 13:48:55.071913 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.071664    4530 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 13:48:55.079256 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079218    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" volumeName="kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 13:48:55.079382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079261    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fafdd535-5a88-4207-b2a4-cc43bf43bd17" volumeName="kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2" seLinuxMountContext=""
Oct 30 13:48:55.079382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079277    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="055c8a12-fcb0-440d-bacc-20dd67b2ce02" volumeName="kubernetes.io/projected/055c8a12-fcb0-440d-bacc-20dd67b2ce02-kube-api-access-6k6s7" seLinuxMountContext=""
Oct 30 13:48:55.079382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079291    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51c2e87b-ea89-48dd-9dd0-d6c674e0f837" volumeName="kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 13:48:55.079382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079303    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9acb4199-6d58-4d3d-86dc-96326517baf8" volumeName="kubernetes.io/projected/9acb4199-6d58-4d3d-86dc-96326517baf8-kube-api-access-lz5fm" seLinuxMountContext=""
Oct 30 13:48:55.079382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079313    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4819261-bdd9-419e-abd3-15fa171de861" volumeName="kubernetes.io/configmap/b4819261-bdd9-419e-abd3-15fa171de861-config-volume" seLinuxMountContext=""
Oct 30 13:48:55.079382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079324    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62977d18-4b0e-4e1e-ba64-bcda9ca2095e" volumeName="kubernetes.io/projected/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-kube-api-access-kfk95" seLinuxMountContext=""
Oct 30 13:48:55.079382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079337    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd2311e-929b-4d5a-9857-47c0500c3be1" volumeName="kubernetes.io/secret/6bd2311e-929b-4d5a-9857-47c0500c3be1-cert" seLinuxMountContext=""
Oct 30 13:48:55.079382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079352    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4819261-bdd9-419e-abd3-15fa171de861" volumeName="kubernetes.io/projected/b4819261-bdd9-419e-abd3-15fa171de861-kube-api-access-dkn6d" seLinuxMountContext=""
Oct 30 13:48:55.079382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079363    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f44206-dd53-4dae-b68c-84a8317227b5" volumeName="kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-service-ca-bundle" seLinuxMountContext=""
Oct 30 13:48:55.079382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079374    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f44206-dd53-4dae-b68c-84a8317227b5" volumeName="kubernetes.io/projected/c6f44206-dd53-4dae-b68c-84a8317227b5-kube-api-access-swft4" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079386    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08311d3a-d790-4761-96f0-e9a6b336e913" volumeName="kubernetes.io/configmap/08311d3a-d790-4761-96f0-e9a6b336e913-serviceca" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079400    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b3dcf54-ba8f-4080-9666-cb41a151ccad" volumeName="kubernetes.io/empty-dir/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-tuned" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079413    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" volumeName="kubernetes.io/projected/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-kube-api-access-xpxql" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079429    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4819261-bdd9-419e-abd3-15fa171de861" volumeName="kubernetes.io/empty-dir/b4819261-bdd9-419e-abd3-15fa171de861-tmp-dir" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079441    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2" volumeName="kubernetes.io/configmap/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2-konnectivity-ca" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079453    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0bbec59-52f9-4dc6-b24b-169f666c08ae" volumeName="kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079465    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="055c8a12-fcb0-440d-bacc-20dd67b2ce02" volumeName="kubernetes.io/configmap/055c8a12-fcb0-440d-bacc-20dd67b2ce02-iptables-alerter-script" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079476    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="375bd1fd-307b-4a6b-b6be-99b492065982" volumeName="kubernetes.io/projected/375bd1fd-307b-4a6b-b6be-99b492065982-kube-api-access-tdwg4" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079505    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51c2e87b-ea89-48dd-9dd0-d6c674e0f837" volumeName="kubernetes.io/empty-dir/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-data-volume" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079517    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51c2e87b-ea89-48dd-9dd0-d6c674e0f837" volumeName="kubernetes.io/projected/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-api-access-2tz4h" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079528    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62977d18-4b0e-4e1e-ba64-bcda9ca2095e" volumeName="kubernetes.io/empty-dir/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-tmp-dir" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079558    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" volumeName="kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079571    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc411f7-86c3-4d69-8f20-af096604e14a" volumeName="kubernetes.io/secret/bbc411f7-86c3-4d69-8f20-af096604e14a-signing-key" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079583    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f44206-dd53-4dae-b68c-84a8317227b5" volumeName="kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079598    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc411f7-86c3-4d69-8f20-af096604e14a" volumeName="kubernetes.io/configmap/bbc411f7-86c3-4d69-8f20-af096604e14a-signing-cabundle" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079611    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b3dcf54-ba8f-4080-9666-cb41a151ccad" volumeName="kubernetes.io/empty-dir/0b3dcf54-ba8f-4080-9666-cb41a151ccad-tmp" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079623    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b3dcf54-ba8f-4080-9666-cb41a151ccad" volumeName="kubernetes.io/projected/0b3dcf54-ba8f-4080-9666-cb41a151ccad-kube-api-access-rp28b" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079660    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" volumeName="kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cni-binary-copy" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079671    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4819261-bdd9-419e-abd3-15fa171de861" volumeName="kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079683    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc411f7-86c3-4d69-8f20-af096604e14a" volumeName="kubernetes.io/projected/bbc411f7-86c3-4d69-8f20-af096604e14a-kube-api-access-5vcdc" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079696    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f44206-dd53-4dae-b68c-84a8317227b5" volumeName="kubernetes.io/empty-dir/c6f44206-dd53-4dae-b68c-84a8317227b5-archives-path" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079707    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7926b85-83e3-4e81-a403-774ea8daebe1" volumeName="kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-ovnkube-config" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079720    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08311d3a-d790-4761-96f0-e9a6b336e913" volumeName="kubernetes.io/projected/08311d3a-d790-4761-96f0-e9a6b336e913-kube-api-access-9f6hc" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079732    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eba42f5-ebfb-457a-ba41-37ba8bed9e4c" volumeName="kubernetes.io/configmap/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-daemon-config" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079743    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7926b85-83e3-4e81-a403-774ea8daebe1" volumeName="kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079756    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7926b85-83e3-4e81-a403-774ea8daebe1" volumeName="kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-env-overrides" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079780    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7926b85-83e3-4e81-a403-774ea8daebe1" volumeName="kubernetes.io/projected/d7926b85-83e3-4e81-a403-774ea8daebe1-kube-api-access-8zvkx" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079791    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7926b85-83e3-4e81-a403-774ea8daebe1" volumeName="kubernetes.io/secret/d7926b85-83e3-4e81-a403-774ea8daebe1-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079807    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eba42f5-ebfb-457a-ba41-37ba8bed9e4c" volumeName="kubernetes.io/configmap/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-cni-binary-copy" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079819    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eba42f5-ebfb-457a-ba41-37ba8bed9e4c" volumeName="kubernetes.io/projected/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-kube-api-access-2rnlm" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079830    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd2311e-929b-4d5a-9857-47c0500c3be1" volumeName="kubernetes.io/projected/6bd2311e-929b-4d5a-9857-47c0500c3be1-kube-api-access-qlxg8" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079845    4530 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2" volumeName="kubernetes.io/secret/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2-agent-certs" seLinuxMountContext=""
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079860    4530 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 13:48:55.079897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.079867    4530 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 13:48:55.081327 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.080033    4530 factory.go:153] Registering CRI-O factory
Oct 30 13:48:55.081327 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.080050    4530 factory.go:223] Registration of the crio container factory successfully
Oct 30 13:48:55.081327 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.080106    4530 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 13:48:55.081327 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.080117    4530 factory.go:55] Registering systemd factory
Oct 30 13:48:55.081327 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.080125    4530 factory.go:223] Registration of the systemd container factory successfully
Oct 30 13:48:55.081327 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.080142    4530 factory.go:103] Registering Raw factory
Oct 30 13:48:55.081327 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.080155    4530 manager.go:1199] Started watching for new ooms in manager
Oct 30 13:48:55.081327 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.080480    4530 manager.go:319] Starting recovery of all containers
Oct 30 13:48:55.105396 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.105365    4530 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 13:48:55.106746 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.106725    4530 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 13:48:55.106746 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.106742    4530 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 13:48:55.106900 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.106757    4530 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 13:48:55.106900 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:55.106790    4530 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 13:48:55.121574 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.121530    4530 generic.go:356] "Generic (PLEG): container finished" podID="b30ac2208b235a46404847e29afd4a13" containerID="4d6a31e8a112d600d96c419998a74d21088ed3a05fdd7c9819e1b34a97965946" exitCode=0
Oct 30 13:48:55.130931 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.130904    4530 generic.go:356] "Generic (PLEG): container finished" podID="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" containerID="59a86eb5be06241c11f43f93b7fb262c303faacd8e0b3789c8f4278f6f6100e8" exitCode=0
Oct 30 13:48:55.130931 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.130931    4530 generic.go:356] "Generic (PLEG): container finished" podID="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" containerID="8767c7145b2b6cd50a2dd8d1280bc829e3e73575745798fe842253ae52476491" exitCode=0
Oct 30 13:48:55.131145 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.130942    4530 generic.go:356] "Generic (PLEG): container finished" podID="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" containerID="2cc7ab8e574457a5d425090c0db23fe0cd349eaf67ce8bcaf5039ee810a13366" exitCode=0
Oct 30 13:48:55.131145 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.130951    4530 generic.go:356] "Generic (PLEG): container finished" podID="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" containerID="bffce27f46f1de025fa3e5d3b9ec636a60b56437f356200209ead96b6e5d07aa" exitCode=0
Oct 30 13:48:55.131145 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.130959    4530 generic.go:356] "Generic (PLEG): container finished" podID="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" containerID="fe143a67765015f2c6fd5669cb3b3dc00cb0852f63b5a53c64bec6d18328478f" exitCode=0
Oct 30 13:48:55.131145 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.130966    4530 generic.go:356] "Generic (PLEG): container finished" podID="b37f6a9a-e81e-40a1-afd4-2726e0bf2614" containerID="4caa023f224a846f660c150030e1463a156e1ffd286672fcda8dc46235e67cec" exitCode=0
Oct 30 13:48:55.131370 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.131209    4530 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:48:55.131419 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.131376    4530 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 13:48:55.145262 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.145239    4530 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:48:55.145581 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.145561    4530 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:48:55.145692 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.145658    4530 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:48:55.149234 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.149218    4530 manager.go:324] Recovery completed
Oct 30 13:48:55.164351 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.164302    4530 cpu_manager.go:226] "Starting" policy="none"
Oct 30 13:48:55.164351 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.164316    4530 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 13:48:55.164351 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.164335    4530 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 13:48:55.164574 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.164513    4530 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 13:48:55.164574 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.164529    4530 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 13:48:55.164574 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.164567    4530 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 13:48:55.164722 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.164576    4530 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 13:48:55.164722 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.164583    4530 policy_none.go:50] "Start"
Oct 30 13:48:55.164722 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.164595    4530 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 13:48:55.164722 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.164608    4530 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 13:48:55.165241 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.165221    4530 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 13:48:55.165241 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.165239    4530 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 13:48:55.165351 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.165247    4530 policy_none.go:44] "Start"
Oct 30 13:48:55.169227 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.169201    4530 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 13:48:55.171773 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.171758    4530 manager.go:341] "Starting Device Plugin manager"
Oct 30 13:48:55.171849 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:55.171784    4530 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 13:48:55.171849 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.171796    4530 server.go:87] "Starting device plugin registration server"
Oct 30 13:48:55.172063 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.172050    4530 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 13:48:55.172121 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.172061    4530 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 13:48:55.172177 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.172129    4530 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 13:48:55.172246 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.172217    4530 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 13:48:55.172301 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.172249    4530 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 13:48:55.172301 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.172254    4530 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 13:48:55.172397 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.172280    4530 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T13:48:55.172242188Z","UUID":"27c81cbe-abd1-489e-aad1-6446e6355845","Handler":null,"Name":"","Endpoint":""}
Oct 30 13:48:55.173584 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:55.173520    4530 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 13:48:55.173584 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.173566    4530 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 13:48:55.173584 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.173586    4530 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 13:48:55.207409 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.207372    4530 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"]
Oct 30 13:48:55.207807 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.207780    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" event={"ID":"b30ac2208b235a46404847e29afd4a13","Type":"ContainerStarted","Data":"0be6eade63d1a73ecf40a4aa995a4038c704e5a72460395c1dd3f8dc4d570ff0"}
Oct 30 13:48:55.207879 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.207814    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" event={"ID":"b30ac2208b235a46404847e29afd4a13","Type":"ContainerDied","Data":"4d6a31e8a112d600d96c419998a74d21088ed3a05fdd7c9819e1b34a97965946"}
Oct 30 13:48:55.207879 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.207829    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" event={"ID":"b30ac2208b235a46404847e29afd4a13","Type":"ContainerStarted","Data":"1fc3828e972ad5a88dd2594e0093f91696fc979c397cba6d0a48747b9a081914"}
Oct 30 13:48:55.207879 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.207870    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" event={"ID":"e329b8d309e1f8141c44f91d4bb26de8","Type":"ContainerStarted","Data":"232b7d148291e15af4ba0fbcd6ccc0288aa4b530081ece9cdade86dcedb93dea"}
Oct 30 13:48:55.208006 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.207883    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" event={"ID":"e329b8d309e1f8141c44f91d4bb26de8","Type":"ContainerStarted","Data":"c71db214edb721aea8aa75d032c94def95d0604d9ae341a7dc3e163700c4777a"}
Oct 30 13:48:55.208006 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.207891    4530 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.208108 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.208002    4530 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.233574 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.233557    4530 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:48:55.233664 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:55.233606    4530 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-118.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.235028 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.235012    4530 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:48:55.235086 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:55.235046    4530 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.272918 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.272897    4530 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:48:55.273968 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.273947    4530 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:48:55.274030 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.273977    4530 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:48:55.274030 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.273986    4530 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:48:55.274030 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.274013    4530 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.316234 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.316215    4530 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.316293 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.316261    4530 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.332598 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.332577    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b30ac2208b235a46404847e29afd4a13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"b30ac2208b235a46404847e29afd4a13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.332700 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.332605    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b30ac2208b235a46404847e29afd4a13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"b30ac2208b235a46404847e29afd4a13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.332700 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.332624    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e329b8d309e1f8141c44f91d4bb26de8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-118.ec2.internal\" (UID: \"e329b8d309e1f8141c44f91d4bb26de8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.433457 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.433408    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b30ac2208b235a46404847e29afd4a13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"b30ac2208b235a46404847e29afd4a13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.433457 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.433432    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b30ac2208b235a46404847e29afd4a13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"b30ac2208b235a46404847e29afd4a13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.433589 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.433451    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e329b8d309e1f8141c44f91d4bb26de8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-118.ec2.internal\" (UID: \"e329b8d309e1f8141c44f91d4bb26de8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.433589 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.433500    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e329b8d309e1f8141c44f91d4bb26de8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-118.ec2.internal\" (UID: \"e329b8d309e1f8141c44f91d4bb26de8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.433589 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.433526    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b30ac2208b235a46404847e29afd4a13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"b30ac2208b235a46404847e29afd4a13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:55.433589 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:55.433526    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b30ac2208b235a46404847e29afd4a13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"b30ac2208b235a46404847e29afd4a13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:56.052783 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.052757    4530 apiserver.go:52] "Watching apiserver"
Oct 30 13:48:56.103493 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.103464    4530 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 13:48:56.103932 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.103911    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal","openshift-network-operator/iptables-alerter-czxww","kube-system/global-pull-secret-syncer-2w5dd","openshift-dns/node-resolver-cjg7z","openshift-insights/insights-runtime-extractor-pb77c","openshift-service-ca/service-ca-5fffc97bdf-9cs6t","openshift-network-diagnostics/network-check-target-98cgg","openshift-image-registry/node-ca-6pbvr","kube-system/konnectivity-agent-sctxm","openshift-cluster-node-tuning-operator/tuned-qlh2v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht","openshift-multus/network-metrics-daemon-l5x6g","openshift-multus/multus-rjz78","openshift-ingress-canary/ingress-canary-xpz2s","openshift-insights/periodic-gathering-f4lw9-sg475","openshift-multus/multus-additional-cni-plugins-g5xcz","openshift-dns/dns-default-nr98w","openshift-ovn-kubernetes/ovnkube-node-kgvgs"]
Oct 30 13:48:56.107318 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.107297    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:56.114912 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.114889    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.122751 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.122730    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k4gxw\""
Oct 30 13:48:56.122751 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.122733    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jc894\""
Oct 30 13:48:56.123045 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.123027    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 13:48:56.123045 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.123035    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 13:48:56.123045 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.123044    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 13:48:56.123234 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.123079    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 13:48:56.123234 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.123117    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 13:48:56.123234 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.123161    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 13:48:56.123234 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.123203    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 13:48:56.123358 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.123252    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 13:48:56.152404 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.152386    4530 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:56.152522 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.152506    4530 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:56.172764 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.172741    4530 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 13:48:56.180661 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.180640    4530 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:48:56.180766 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:56.180702    4530 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-118.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:56.180766 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.180650    4530 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:48:56.180866 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:56.180787    4530 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"
Oct 30 13:48:56.196021 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196002    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 13:48:56.196124 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196110    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 13:48:56.196332 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196315    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:48:56.196462 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196334    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 13:48:56.196462 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196362    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 13:48:56.196462 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196377    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 13:48:56.196462 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196385    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 13:48:56.196462 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196395    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 13:48:56.196462 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196434    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 13:48:56.196462 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196437    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 13:48:56.196855 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196576    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 13:48:56.196855 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196606    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 13:48:56.196855 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196627    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 13:48:56.196855 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196638    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 13:48:56.196855 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196631    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 13:48:56.196855 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196655    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 13:48:56.196855 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.196674    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 13:48:56.197203 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.197186    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 13:48:56.197416 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.197401    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 13:48:56.197466 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.197430    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 13:48:56.197495 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.197482    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 13:48:56.197529 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.197505    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 13:48:56.197529 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.197505    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 13:48:56.197714 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.197557    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:48:56.197714 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.197572    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 13:48:56.197714 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.197592    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 13:48:56.197714 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.197626    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 13:48:56.197714 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.197669    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 13:48:56.211281 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211241    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 13:48:56.211281 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211273    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 13:48:56.211450 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211373    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 13:48:56.211450 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211385    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 13:48:56.211574 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211560    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d6wsl\""
Oct 30 13:48:56.211976 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211677    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 13:48:56.211976 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211692    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-825mf\""
Oct 30 13:48:56.211976 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211682    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xn5x9\""
Oct 30 13:48:56.211976 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211696    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 13:48:56.211976 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211956    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-smj8j\""
Oct 30 13:48:56.211976 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211970    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5r2q6\""
Oct 30 13:48:56.212221 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211983    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2dbfk\""
Oct 30 13:48:56.212221 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.211996    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b7cls\""
Oct 30 13:48:56.212221 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.212016    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w6ztc\""
Oct 30 13:48:56.212221 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.212052    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-snt8q\""
Oct 30 13:48:56.212221 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.212087    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 13:48:56.212221 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.212189    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7j7wh\""
Oct 30 13:48:56.212221 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.212212    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-697m8\""
Oct 30 13:48:56.212413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.212249    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 13:48:56.212413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.212250    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w8shz\""
Oct 30 13:48:56.212413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.212312    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 13:48:56.212413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.212360    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gw59s\""
Oct 30 13:48:56.212413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.212406    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 13:48:56.212413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.212411    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mxshj\""
Oct 30 13:48:56.212584 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.212474    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s5pv6\""
Oct 30 13:48:56.236347 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236327    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.236410 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236353    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4819261-bdd9-419e-abd3-15fa171de861-tmp-dir\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:56.236410 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236371    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkn6d\" (UniqueName: \"kubernetes.io/projected/b4819261-bdd9-419e-abd3-15fa171de861-kube-api-access-dkn6d\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:56.236410 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236391    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rnlm\" (UniqueName: \"kubernetes.io/projected/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-kube-api-access-2rnlm\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.236517 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236420    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-sys-fs\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.236517 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236454    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bbc411f7-86c3-4d69-8f20-af096604e14a-signing-key\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:56.236609 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236513    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-run-netns\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.236609 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236576    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz5fm\" (UniqueName: \"kubernetes.io/projected/9acb4199-6d58-4d3d-86dc-96326517baf8-kube-api-access-lz5fm\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:56.236700 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236682    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4819261-bdd9-419e-abd3-15fa171de861-tmp-dir\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:56.236740 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236703    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-cnibin\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.236740 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236684    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bbc411f7-86c3-4d69-8f20-af096604e14a-signing-key\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:56.236740 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236725    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-sys\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.236842 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236744    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k6s7\" (UniqueName: \"kubernetes.io/projected/055c8a12-fcb0-440d-bacc-20dd67b2ce02-kube-api-access-6k6s7\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:56.236842 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236775    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-systemd\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.236842 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236834    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-etc-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.236996 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236863    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-env-overrides\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.236996 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236886    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.236996 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.236946    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.237122 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237007    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-cni-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.237122 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237033    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-socket-dir-parent\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.237122 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237080    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-systemd\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.237247 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237155    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-run\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.237247 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237189    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-env-overrides\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.237247 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237202    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:56.237384 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237248    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2-konnectivity-ca\") pod \"konnectivity-agent-sctxm\" (UID: \"efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2\") " pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:56.237384 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237342    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.237441 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237413    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4819261-bdd9-419e-abd3-15fa171de861-metrics-tls\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:56.237499 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237484    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2-konnectivity-ca\") pod \"konnectivity-agent-sctxm\" (UID: \"efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2\") " pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:56.237685 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237672    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-kubelet\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.237721 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237699    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-ovnkube-script-lib\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.237773 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237762    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-lib-modules\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.237805 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237780    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-socket-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.237805 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237799    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4819261-bdd9-419e-abd3-15fa171de861-config-volume\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:56.237898 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237818    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-systemd-units\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.237898 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237834    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.237898 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237836    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-ovnkube-script-lib\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.237898 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237878    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxql\" (UniqueName: \"kubernetes.io/projected/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-kube-api-access-xpxql\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.238051 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.237959    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.238051 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238010    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.238051 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238015    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4819261-bdd9-419e-abd3-15fa171de861-config-volume\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:56.238192 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238067    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08311d3a-d790-4761-96f0-e9a6b336e913-serviceca\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:56.238192 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238166    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6hc\" (UniqueName: \"kubernetes.io/projected/08311d3a-d790-4761-96f0-e9a6b336e913-kube-api-access-9f6hc\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:56.238276 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238195    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b3dcf54-ba8f-4080-9666-cb41a151ccad-tmp\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.238276 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238228    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.238276 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238255    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44fn2\" (UniqueName: \"kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2\") pod \"network-check-target-98cgg\" (UID: \"fafdd535-5a88-4207-b2a4-cc43bf43bd17\") " pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:56.238390 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238283    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b3dcf54-ba8f-4080-9666-cb41a151ccad-tmp\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.238390 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238311    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:56.238390 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238349    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6f44206-dd53-4dae-b68c-84a8317227b5-archives-path\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:56.238473 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238452    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swft4\" (UniqueName: \"kubernetes.io/projected/c6f44206-dd53-4dae-b68c-84a8317227b5-kube-api-access-swft4\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:56.238510 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238488    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6f44206-dd53-4dae-b68c-84a8317227b5-archives-path\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:56.238556 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238518    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.238607 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238582    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfk95\" (UniqueName: \"kubernetes.io/projected/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-kube-api-access-kfk95\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:56.238672 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238620    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysconfig\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.238716 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238667    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-kubernetes\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.238768 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238716    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-tuned\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.238768 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238720    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/08311d3a-d790-4761-96f0-e9a6b336e913-serviceca\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:56.238768 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238756    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwg4\" (UniqueName: \"kubernetes.io/projected/375bd1fd-307b-4a6b-b6be-99b492065982-kube-api-access-tdwg4\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.238862 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238788    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-multus-certs\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.238862 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238809    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvkx\" (UniqueName: \"kubernetes.io/projected/d7926b85-83e3-4e81-a403-774ea8daebe1-kube-api-access-8zvkx\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.238862 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238830    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-system-cni-dir\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.238977 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238846    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-hosts-file\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:56.238977 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238887    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:56.238977 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238911    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-var-lib-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.238977 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238918    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-tuned\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.238977 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.238946    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-node-log\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.239136 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239062    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-tmp-dir\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:56.239136 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239091    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055c8a12-fcb0-440d-bacc-20dd67b2ce02-host-slash\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:56.239136 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239109    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2-agent-certs\") pod \"konnectivity-agent-sctxm\" (UID: \"efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2\") " pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:56.239136 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239125    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0bbec59-52f9-4dc6-b24b-169f666c08ae-original-pull-secret\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:56.239252 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239140    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cni-binary-copy\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.239252 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239237    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-tmp-dir\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:56.239312 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239278    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-trusted-ca-bundle\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:56.239312 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239283    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2-agent-certs\") pod \"konnectivity-agent-sctxm\" (UID: \"efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2\") " pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:48:56.239375 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239360    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-crio-socket\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.239409 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239401    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cni-binary-copy\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.239462 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239446    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tz4h\" (UniqueName: \"kubernetes.io/projected/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-api-access-2tz4h\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.239496 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239471    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-cni-multus\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.239496 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239486    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/055c8a12-fcb0-440d-bacc-20dd67b2ce02-iptables-alerter-script\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:56.239568 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239505    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0bbec59-52f9-4dc6-b24b-169f666c08ae-kubelet-config\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:56.239600 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239568    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-trusted-ca-bundle\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:56.239627 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239605    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.239661 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239634    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.239698 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239657    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08311d3a-d790-4761-96f0-e9a6b336e913-host\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:56.239698 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239679    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-netns\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.239750 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239710    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bd2311e-929b-4d5a-9857-47c0500c3be1-cert\") pod \"ingress-canary-xpz2s\" (UID: \"6bd2311e-929b-4d5a-9857-47c0500c3be1\") " pod="openshift-ingress-canary/ingress-canary-xpz2s"
Oct 30 13:48:56.239777 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239745    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/055c8a12-fcb0-440d-bacc-20dd67b2ce02-iptables-alerter-script\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:56.239777 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239752    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vcdc\" (UniqueName: \"kubernetes.io/projected/bbc411f7-86c3-4d69-8f20-af096604e14a-kube-api-access-5vcdc\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:56.239848 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239837    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-log-socket\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.239895 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239881    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cnibin\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.239927 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239907    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bd2311e-929b-4d5a-9857-47c0500c3be1-cert\") pod \"ingress-canary-xpz2s\" (UID: \"6bd2311e-929b-4d5a-9857-47c0500c3be1\") " pod="openshift-ingress-canary/ingress-canary-xpz2s"
Oct 30 13:48:56.239927 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239909    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-etc-kubernetes\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.239982 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239939    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-modprobe-d\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.239982 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239966    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysctl-conf\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.240037 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.239994    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-registration-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.240037 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240016    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-etc-selinux\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.240037 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240031    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-ovn\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.240116 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240045    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-conf-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.240116 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240065    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0bbec59-52f9-4dc6-b24b-169f666c08ae-dbus\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:56.240116 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240101    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bbc411f7-86c3-4d69-8f20-af096604e14a-signing-cabundle\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:56.240192 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240125    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.240192 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240141    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-cni-bin\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.240192 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240157    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-cni-netd\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.240192 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240172    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7926b85-83e3-4e81-a403-774ea8daebe1-ovn-node-metrics-cert\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.240303 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240200    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-data-volume\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.240303 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240268    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-data-volume\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.240303 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240272    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bbc411f7-86c3-4d69-8f20-af096604e14a-signing-cabundle\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:56.240382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240306    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-daemon-config\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.240382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240333    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-ovnkube-config\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.240382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240349    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-service-ca-bundle\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:56.240458 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240402    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-system-cni-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.240496 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240467    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7926b85-83e3-4e81-a403-774ea8daebe1-ovn-node-metrics-cert\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.240496 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240483    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-service-ca-bundle\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:56.240571 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240496    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-cni-binary-copy\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.240571 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240523    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-kubelet\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.240571 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240524    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d7926b85-83e3-4e81-a403-774ea8daebe1-ovnkube-config\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.240688 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240577    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-hostroot\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.240688 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240597    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-daemon-config\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.240688 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240623    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp28b\" (UniqueName: \"kubernetes.io/projected/0b3dcf54-ba8f-4080-9666-cb41a151ccad-kube-api-access-rp28b\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.240688 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240643    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-device-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.240688 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240659    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxg8\" (UniqueName: \"kubernetes.io/projected/6bd2311e-929b-4d5a-9857-47c0500c3be1-kube-api-access-qlxg8\") pod \"ingress-canary-xpz2s\" (UID: \"6bd2311e-929b-4d5a-9857-47c0500c3be1\") " pod="openshift-ingress-canary/ingress-canary-xpz2s"
Oct 30 13:48:56.240688 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240677    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-slash\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.240855 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240659    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-cni-binary-copy\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.240905 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240891    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-k8s-cni-cncf-io\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.240934 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240913    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-cni-bin\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.240934 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240927    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysctl-d\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.240997 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240956    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-var-lib-kubelet\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.240997 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.240981    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-host\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.265880 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.265860    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkn6d\" (UniqueName: \"kubernetes.io/projected/b4819261-bdd9-419e-abd3-15fa171de861-kube-api-access-dkn6d\") pod \"dns-default-nr98w\" (UID: \"b4819261-bdd9-419e-abd3-15fa171de861\") " pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:56.266051 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.266035    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tz4h\" (UniqueName: \"kubernetes.io/projected/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-kube-api-access-2tz4h\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.266577 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.266559    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rnlm\" (UniqueName: \"kubernetes.io/projected/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-kube-api-access-2rnlm\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.266637 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.266584    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vcdc\" (UniqueName: \"kubernetes.io/projected/bbc411f7-86c3-4d69-8f20-af096604e14a-kube-api-access-5vcdc\") pod \"service-ca-5fffc97bdf-9cs6t\" (UID: \"bbc411f7-86c3-4d69-8f20-af096604e14a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9cs6t"
Oct 30 13:48:56.266909 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.266893    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvkx\" (UniqueName: \"kubernetes.io/projected/d7926b85-83e3-4e81-a403-774ea8daebe1-kube-api-access-8zvkx\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.270134 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.267788    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfk95\" (UniqueName: \"kubernetes.io/projected/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-kube-api-access-kfk95\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:56.270134 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.268165    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swft4\" (UniqueName: \"kubernetes.io/projected/c6f44206-dd53-4dae-b68c-84a8317227b5-kube-api-access-swft4\") pod \"periodic-gathering-f4lw9-sg475\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") " pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:48:56.270134 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.268478    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6hc\" (UniqueName: \"kubernetes.io/projected/08311d3a-d790-4761-96f0-e9a6b336e913-kube-api-access-9f6hc\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:56.270134 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.268609    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz5fm\" (UniqueName: \"kubernetes.io/projected/9acb4199-6d58-4d3d-86dc-96326517baf8-kube-api-access-lz5fm\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:56.270134 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.268759    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp28b\" (UniqueName: \"kubernetes.io/projected/0b3dcf54-ba8f-4080-9666-cb41a151ccad-kube-api-access-rp28b\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.270939 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.270917    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxql\" (UniqueName: \"kubernetes.io/projected/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-kube-api-access-xpxql\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.271010 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.270984    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k6s7\" (UniqueName: \"kubernetes.io/projected/055c8a12-fcb0-440d-bacc-20dd67b2ce02-kube-api-access-6k6s7\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:56.271095 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.271073    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwg4\" (UniqueName: \"kubernetes.io/projected/375bd1fd-307b-4a6b-b6be-99b492065982-kube-api-access-tdwg4\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.271259 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.271244    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxg8\" (UniqueName: \"kubernetes.io/projected/6bd2311e-929b-4d5a-9857-47c0500c3be1-kube-api-access-qlxg8\") pod \"ingress-canary-xpz2s\" (UID: \"6bd2311e-929b-4d5a-9857-47c0500c3be1\") " pod="openshift-ingress-canary/ingress-canary-xpz2s"
Oct 30 13:48:56.272443 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.272427    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44fn2\" (UniqueName: \"kubernetes.io/projected/fafdd535-5a88-4207-b2a4-cc43bf43bd17-kube-api-access-44fn2\") pod \"network-check-target-98cgg\" (UID: \"fafdd535-5a88-4207-b2a4-cc43bf43bd17\") " pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:48:56.341428 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341382    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-cni-multus\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.341428 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341407    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0bbec59-52f9-4dc6-b24b-169f666c08ae-kubelet-config\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:56.341428 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341421    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341436    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341451    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08311d3a-d790-4761-96f0-e9a6b336e913-host\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341467    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-netns\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341495    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0bbec59-52f9-4dc6-b24b-169f666c08ae-kubelet-config\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341495    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-cni-multus\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341514    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-netns\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341524    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341514    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341535    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-log-socket\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341582    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-log-socket\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341569    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08311d3a-d790-4761-96f0-e9a6b336e913-host\") pod \"node-ca-6pbvr\" (UID: \"08311d3a-d790-4761-96f0-e9a6b336e913\") " pod="openshift-image-registry/node-ca-6pbvr"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341604    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cnibin\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.341620 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341624    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-etc-kubernetes\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341637    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-modprobe-d\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341651    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysctl-conf\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341670    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-etc-kubernetes\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341669    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-registration-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341702    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-cnibin\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341720    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-etc-selinux\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341756    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-modprobe-d\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341786    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-registration-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341786    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysctl-conf\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341785    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-etc-selinux\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341808    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-ovn\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341844    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-ovn\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341877    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-conf-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341902    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0bbec59-52f9-4dc6-b24b-169f666c08ae-dbus\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341914    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-conf-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341948    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341954    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0bbec59-52f9-4dc6-b24b-169f666c08ae-dbus\") pod \"global-pull-secret-syncer-2w5dd\" (UID: \"f0bbec59-52f9-4dc6-b24b-169f666c08ae\") " pod="kube-system/global-pull-secret-syncer-2w5dd"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341972    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-cni-bin\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.341989    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342003    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-cni-netd\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342032    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-system-cni-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342005    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-cni-bin\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342055    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-kubelet\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342084    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-cni-netd\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342102    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-system-cni-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342109    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-hostroot\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342101    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-kubelet\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342128    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-device-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342144    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-slash\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342158    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-k8s-cni-cncf-io\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342168    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-device-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342174    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-cni-bin\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342193    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysctl-d\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.342170 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342193    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-k8s-cni-cncf-io\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342145    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-hostroot\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342206    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-var-lib-kubelet\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342206    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-var-lib-cni-bin\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342204    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-slash\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342229    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-host\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342245    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342246    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-var-lib-kubelet\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342275    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-sys-fs\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342285    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342297    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysctl-d\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342300    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-run-netns\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342309    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-host\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342324    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-cnibin\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342331    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-sys-fs\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342345    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-sys\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342370    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-run-netns\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342387    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-systemd\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342396    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-sys\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342396    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-cnibin\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342402    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-etc-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342420    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-etc-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342424    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-run-systemd\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342435    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342463    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-cni-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342486    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-socket-dir-parent\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342528    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342557    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-socket-dir-parent\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342561    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-systemd\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342584    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-systemd\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342595    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-run\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342620    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-kubelet\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342562    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-multus-cni-dir\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342643    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-lib-modules\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342673    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-run\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342678    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-socket-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342682    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-host-kubelet\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342707    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-systemd-units\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342749    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-lib-modules\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342766    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/375bd1fd-307b-4a6b-b6be-99b492065982-socket-dir\") pod \"aws-ebs-csi-driver-node-jk2ht\" (UID: \"375bd1fd-307b-4a6b-b6be-99b492065982\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk2ht"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342781    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-systemd-units\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342797    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342818    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342834    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysconfig\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342847    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-kubernetes\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342870    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-multus-certs\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342894    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-system-cni-dir\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342900    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-sysconfig\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342917    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-hosts-file\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342929    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b3dcf54-ba8f-4080-9666-cb41a151ccad-etc-kubernetes\") pod \"tuned-qlh2v\" (UID: \"0b3dcf54-ba8f-4080-9666-cb41a151ccad\") " pod="openshift-cluster-node-tuning-operator/tuned-qlh2v"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342939    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b37f6a9a-e81e-40a1-afd4-2726e0bf2614-system-cni-dir\") pod \"multus-additional-cni-plugins-g5xcz\" (UID: \"b37f6a9a-e81e-40a1-afd4-2726e0bf2614\") " pod="openshift-multus/multus-additional-cni-plugins-g5xcz"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342942    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-var-lib-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:56.342934    4530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342969    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-node-log\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342978    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62977d18-4b0e-4e1e-ba64-bcda9ca2095e-hosts-file\") pod \"node-resolver-cjg7z\" (UID: \"62977d18-4b0e-4e1e-ba64-bcda9ca2095e\") " pod="openshift-dns/node-resolver-cjg7z"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342993    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055c8a12-fcb0-440d-bacc-20dd67b2ce02-host-slash\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.343008    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-var-lib-openvswitch\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.343016    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d7926b85-83e3-4e81-a403-774ea8daebe1-node-log\") pod \"ovnkube-node-kgvgs\" (UID: \"d7926b85-83e3-4e81-a403-774ea8daebe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.342950    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1eba42f5-ebfb-457a-ba41-37ba8bed9e4c-host-run-multus-certs\") pod \"multus-rjz78\" (UID: \"1eba42f5-ebfb-457a-ba41-37ba8bed9e4c\") " pod="openshift-multus/multus-rjz78"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.343033    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055c8a12-fcb0-440d-bacc-20dd67b2ce02-host-slash\") pod \"iptables-alerter-czxww\" (UID: \"055c8a12-fcb0-440d-bacc-20dd67b2ce02\") " pod="openshift-network-operator/iptables-alerter-czxww"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:56.343028    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:56.843009687 +0000 UTC m=+1.874908629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.343060    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-crio-socket\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.343119    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-crio-socket\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.343116    4530 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 13:48:56.345197 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.345125    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acb4199-6d58-4d3d-86dc-96326517baf8-metrics-certs\") pod \"network-metrics-daemon-l5x6g\" (UID: \"9acb4199-6d58-4d3d-86dc-96326517baf8\") " pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:56.407776 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.407758    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5x6g"
Oct 30 13:48:56.415326 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.415312    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:48:56.731198 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.731173    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l5x6g"]
Oct 30 13:48:56.732586 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:48:56.732553    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9acb4199_6d58_4d3d_86dc_96326517baf8.slice/crio-d326bf17872fb14afe414802cb32ae151e50688cf350194c3cb5012631f4b54f WatchSource:0}: Error finding container d326bf17872fb14afe414802cb32ae151e50688cf350194c3cb5012631f4b54f: Status 404 returned error can't find the container with id d326bf17872fb14afe414802cb32ae151e50688cf350194c3cb5012631f4b54f
Oct 30 13:48:56.847349 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:56.847315    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:56.847491 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:56.847443    4530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:56.847571 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:56.847509    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:57.84749024 +0000 UTC m=+2.879389159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:57.155589 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:57.155564    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nr98w" event={"ID":"b4819261-bdd9-419e-abd3-15fa171de861","Type":"ContainerStarted","Data":"7aaedc05af9652fc6c37d2f3d2a5db9151d68707370a057dbd492e371b33133e"}
Oct 30 13:48:57.156802 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:57.156776    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l5x6g" event={"ID":"9acb4199-6d58-4d3d-86dc-96326517baf8","Type":"ContainerStarted","Data":"d326bf17872fb14afe414802cb32ae151e50688cf350194c3cb5012631f4b54f"}
Oct 30 13:48:57.854412 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:57.854378    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:57.854597 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:57.854490    4530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:57.854597 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:57.854560    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:59.854529829 +0000 UTC m=+4.886428746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:58.161594 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:58.161563    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nr98w" event={"ID":"b4819261-bdd9-419e-abd3-15fa171de861","Type":"ContainerStarted","Data":"1f412435cc3372783d5fcc1eb66a7fdfd42b0bc15509cda947806b7b8c021a1d"}
Oct 30 13:48:58.161983 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:58.161667    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nr98w"
Oct 30 13:48:58.163007 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:58.162984    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l5x6g" event={"ID":"9acb4199-6d58-4d3d-86dc-96326517baf8","Type":"ContainerStarted","Data":"815edd51a0f4fc501e8940486be710097358b090535015a918b115adf9e54c2a"}
Oct 30 13:48:59.167947 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:59.167916    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l5x6g" event={"ID":"9acb4199-6d58-4d3d-86dc-96326517baf8","Type":"ContainerStarted","Data":"3e0ddade82139513cf4162ac8a6da4194e294ee00c645f28f5aa053201ba26bd"}
Oct 30 13:48:59.708331 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:59.708307    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:48:59.867401 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:48:59.867371    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:48:59.867555 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:59.867482    4530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 13:48:59.867555 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:48:59.867530    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls podName:51c2e87b-ea89-48dd-9dd0-d6c674e0f837 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:03.86751832 +0000 UTC m=+8.899417237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pb77c" (UID: "51c2e87b-ea89-48dd-9dd0-d6c674e0f837") : secret "insights-runtime-extractor-tls" not found
Oct 30 13:49:03.560861 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:03.560833    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:49:03.561234 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:03.561087    4530 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:49:03.890169 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:03.890094    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:49:03.892241 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:03.892220    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c2e87b-ea89-48dd-9dd0-d6c674e0f837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pb77c\" (UID: \"51c2e87b-ea89-48dd-9dd0-d6c674e0f837\") " pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:49:03.916218 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:03.916191    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pb77c"
Oct 30 13:49:04.207403 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:04.207376    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pb77c"]
Oct 30 13:49:04.209145 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:49:04.209115    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c2e87b_ea89_48dd_9dd0_d6c674e0f837.slice/crio-12c549033d3fc0429121ad36111cbb77c43f3983f786e3f84977d66b1a9de26a WatchSource:0}: Error finding container 12c549033d3fc0429121ad36111cbb77c43f3983f786e3f84977d66b1a9de26a: Status 404 returned error can't find the container with id 12c549033d3fc0429121ad36111cbb77c43f3983f786e3f84977d66b1a9de26a
Oct 30 13:49:04.723880 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:04.723852    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:49:05.187929 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:05.187901    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pb77c" event={"ID":"51c2e87b-ea89-48dd-9dd0-d6c674e0f837","Type":"ContainerStarted","Data":"7f7f84fe9d86abb2d60ef9c89c3223f622a676791ad63007cae87c05d73435aa"}
Oct 30 13:49:05.188047 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:05.187949    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pb77c" event={"ID":"51c2e87b-ea89-48dd-9dd0-d6c674e0f837","Type":"ContainerStarted","Data":"88e0095613dadc21687ce875cdaa45f93025cc38c3c39c8b89d951067c4c4342"}
Oct 30 13:49:05.188047 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:05.187991    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pb77c" event={"ID":"51c2e87b-ea89-48dd-9dd0-d6c674e0f837","Type":"ContainerStarted","Data":"12c549033d3fc0429121ad36111cbb77c43f3983f786e3f84977d66b1a9de26a"}
Oct 30 13:49:06.411955 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.411925    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4"]
Oct 30 13:49:06.413914 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.413898    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4"
Oct 30 13:49:06.425981 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.425962    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-cls76"]
Oct 30 13:49:06.427662 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.427648    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-cls76"
Oct 30 13:49:06.435255 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.435195    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4"]
Oct 30 13:49:06.435699 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.435682    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-cls76"]
Oct 30 13:49:06.448489 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.448467    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cklnl\""
Oct 30 13:49:06.448625 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.448579    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 13:49:06.450310 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.450290    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 13:49:06.450396 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.450312    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 13:49:06.505594 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.505530    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5c1b1ee0-e5fc-4ce7-b3ad-d9880a5db41f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-zq9x4\" (UID: \"5c1b1ee0-e5fc-4ce7-b3ad-d9880a5db41f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4"
Oct 30 13:49:06.510019 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.510001    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fb64c5687-zhdxl"]
Oct 30 13:49:06.511890 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.511873    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.547944 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.547919    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 13:49:06.547944 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.547940    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 13:49:06.548098 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.547955    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t9cdd\""
Oct 30 13:49:06.548098 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.548080    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 13:49:06.548190 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.548131    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 13:49:06.548190 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.548184    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-xlvfw\""
Oct 30 13:49:06.548616 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.548599    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fb64c5687-zhdxl"]
Oct 30 13:49:06.606382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.606358    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9dc9ea80-315c-4d96-b62b-5089226a9f65-installation-pull-secrets\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.606460 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.606393    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9dc9ea80-315c-4d96-b62b-5089226a9f65-ca-trust-extracted\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.606460 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.606424    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9dc9ea80-315c-4d96-b62b-5089226a9f65-trusted-ca\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.606534 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.606499    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12678c60-0b95-404d-aa2d-a10c0ccf91f4-tmp\") pod \"downloads-6dd789c8b7-cls76\" (UID: \"12678c60-0b95-404d-aa2d-a10c0ccf91f4\") " pod="openshift-console/downloads-6dd789c8b7-cls76"
Oct 30 13:49:06.606605 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.606593    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9dc9ea80-315c-4d96-b62b-5089226a9f65-image-registry-private-configuration\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.606704 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.606688    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5c1b1ee0-e5fc-4ce7-b3ad-d9880a5db41f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-zq9x4\" (UID: \"5c1b1ee0-e5fc-4ce7-b3ad-d9880a5db41f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4"
Oct 30 13:49:06.606735 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.606716    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrbv\" (UniqueName: \"kubernetes.io/projected/12678c60-0b95-404d-aa2d-a10c0ccf91f4-kube-api-access-lqrbv\") pod \"downloads-6dd789c8b7-cls76\" (UID: \"12678c60-0b95-404d-aa2d-a10c0ccf91f4\") " pod="openshift-console/downloads-6dd789c8b7-cls76"
Oct 30 13:49:06.606766 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.606734    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9dc9ea80-315c-4d96-b62b-5089226a9f65-registry-certificates\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.606800 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.606764    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9dc9ea80-315c-4d96-b62b-5089226a9f65-bound-sa-token\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.606839 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.606822    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dc9ea80-315c-4d96-b62b-5089226a9f65-registry-tls\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.606885 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.606871    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tz2\" (UniqueName: \"kubernetes.io/projected/9dc9ea80-315c-4d96-b62b-5089226a9f65-kube-api-access-h5tz2\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.608979 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.608955    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5c1b1ee0-e5fc-4ce7-b3ad-d9880a5db41f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-zq9x4\" (UID: \"5c1b1ee0-e5fc-4ce7-b3ad-d9880a5db41f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4"
Oct 30 13:49:06.707800 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.707778    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrbv\" (UniqueName: \"kubernetes.io/projected/12678c60-0b95-404d-aa2d-a10c0ccf91f4-kube-api-access-lqrbv\") pod \"downloads-6dd789c8b7-cls76\" (UID: \"12678c60-0b95-404d-aa2d-a10c0ccf91f4\") " pod="openshift-console/downloads-6dd789c8b7-cls76"
Oct 30 13:49:06.707905 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.707804    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9dc9ea80-315c-4d96-b62b-5089226a9f65-registry-certificates\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.707905 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.707819    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9dc9ea80-315c-4d96-b62b-5089226a9f65-bound-sa-token\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.707905 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.707836    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dc9ea80-315c-4d96-b62b-5089226a9f65-registry-tls\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.707905 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.707853    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tz2\" (UniqueName: \"kubernetes.io/projected/9dc9ea80-315c-4d96-b62b-5089226a9f65-kube-api-access-h5tz2\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.707905 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.707879    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9dc9ea80-315c-4d96-b62b-5089226a9f65-installation-pull-secrets\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.707905 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.707901    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9dc9ea80-315c-4d96-b62b-5089226a9f65-ca-trust-extracted\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.708199 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.707922    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9dc9ea80-315c-4d96-b62b-5089226a9f65-trusted-ca\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.708199 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.707937    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12678c60-0b95-404d-aa2d-a10c0ccf91f4-tmp\") pod \"downloads-6dd789c8b7-cls76\" (UID: \"12678c60-0b95-404d-aa2d-a10c0ccf91f4\") " pod="openshift-console/downloads-6dd789c8b7-cls76"
Oct 30 13:49:06.708199 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.707956    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9dc9ea80-315c-4d96-b62b-5089226a9f65-image-registry-private-configuration\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.708459 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.708432    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12678c60-0b95-404d-aa2d-a10c0ccf91f4-tmp\") pod \"downloads-6dd789c8b7-cls76\" (UID: \"12678c60-0b95-404d-aa2d-a10c0ccf91f4\") " pod="openshift-console/downloads-6dd789c8b7-cls76"
Oct 30 13:49:06.708583 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.708436    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9dc9ea80-315c-4d96-b62b-5089226a9f65-ca-trust-extracted\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.709106 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.709079    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9dc9ea80-315c-4d96-b62b-5089226a9f65-registry-certificates\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.709179 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.709128    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9dc9ea80-315c-4d96-b62b-5089226a9f65-trusted-ca\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.710238 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.710213    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9dc9ea80-315c-4d96-b62b-5089226a9f65-registry-tls\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.710351 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.710330    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9dc9ea80-315c-4d96-b62b-5089226a9f65-image-registry-private-configuration\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.710464 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.710448    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9dc9ea80-315c-4d96-b62b-5089226a9f65-installation-pull-secrets\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.723426 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.723409    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4"
Oct 30 13:49:06.736562 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.736522    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrbv\" (UniqueName: \"kubernetes.io/projected/12678c60-0b95-404d-aa2d-a10c0ccf91f4-kube-api-access-lqrbv\") pod \"downloads-6dd789c8b7-cls76\" (UID: \"12678c60-0b95-404d-aa2d-a10c0ccf91f4\") " pod="openshift-console/downloads-6dd789c8b7-cls76"
Oct 30 13:49:06.736748 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.736728    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tz2\" (UniqueName: \"kubernetes.io/projected/9dc9ea80-315c-4d96-b62b-5089226a9f65-kube-api-access-h5tz2\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.736843 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.736827    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9dc9ea80-315c-4d96-b62b-5089226a9f65-bound-sa-token\") pod \"image-registry-5fb64c5687-zhdxl\" (UID: \"9dc9ea80-315c-4d96-b62b-5089226a9f65\") " pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:06.737333 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.737309    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-cls76"
Oct 30 13:49:06.821022 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:06.820999    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:07.017422 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:07.017397    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4"]
Oct 30 13:49:07.018240 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:49:07.018209    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1b1ee0_e5fc_4ce7_b3ad_d9880a5db41f.slice/crio-24cc19388bdaad98d6f829f600ce7d423170ae6ef4b4e200d785784cea336e75 WatchSource:0}: Error finding container 24cc19388bdaad98d6f829f600ce7d423170ae6ef4b4e200d785784cea336e75: Status 404 returned error can't find the container with id 24cc19388bdaad98d6f829f600ce7d423170ae6ef4b4e200d785784cea336e75
Oct 30 13:49:07.037456 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:07.037434    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-cls76"]
Oct 30 13:49:07.038251 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:49:07.038225    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12678c60_0b95_404d_aa2d_a10c0ccf91f4.slice/crio-296f8d51d7c4f467ffa0d9b21a72edfbcdd40586064b990b982394fb4e4dd710 WatchSource:0}: Error finding container 296f8d51d7c4f467ffa0d9b21a72edfbcdd40586064b990b982394fb4e4dd710: Status 404 returned error can't find the container with id 296f8d51d7c4f467ffa0d9b21a72edfbcdd40586064b990b982394fb4e4dd710
Oct 30 13:49:07.117599 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:07.117575    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fb64c5687-zhdxl"]
Oct 30 13:49:07.118035 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:49:07.118012    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc9ea80_315c_4d96_b62b_5089226a9f65.slice/crio-2372c554dc45f2c2cc60492f0eb022d723ba0f09fd917794e9c46b701d5be042 WatchSource:0}: Error finding container 2372c554dc45f2c2cc60492f0eb022d723ba0f09fd917794e9c46b701d5be042: Status 404 returned error can't find the container with id 2372c554dc45f2c2cc60492f0eb022d723ba0f09fd917794e9c46b701d5be042
Oct 30 13:49:07.195172 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:07.195144    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl" event={"ID":"9dc9ea80-315c-4d96-b62b-5089226a9f65","Type":"ContainerStarted","Data":"84b3876de32ad37ca84e0ab4c7272c18a5716b24604fbae4ba33de6088067fd1"}
Oct 30 13:49:07.195304 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:07.195202    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl" event={"ID":"9dc9ea80-315c-4d96-b62b-5089226a9f65","Type":"ContainerStarted","Data":"2372c554dc45f2c2cc60492f0eb022d723ba0f09fd917794e9c46b701d5be042"}
Oct 30 13:49:07.195304 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:07.195228    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:07.196352 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:07.196325    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-cls76" event={"ID":"12678c60-0b95-404d-aa2d-a10c0ccf91f4","Type":"ContainerStarted","Data":"296f8d51d7c4f467ffa0d9b21a72edfbcdd40586064b990b982394fb4e4dd710"}
Oct 30 13:49:07.202585 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:07.202558    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pb77c" event={"ID":"51c2e87b-ea89-48dd-9dd0-d6c674e0f837","Type":"ContainerStarted","Data":"910f4f2315acc0e8441f6a57af5e386101ff505ea7eb4bc13b97c627729f4914"}
Oct 30 13:49:07.203677 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:07.203651    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4" event={"ID":"5c1b1ee0-e5fc-4ce7-b3ad-d9880a5db41f","Type":"ContainerStarted","Data":"24cc19388bdaad98d6f829f600ce7d423170ae6ef4b4e200d785784cea336e75"}
Oct 30 13:49:07.246610 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:07.246560    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl" podStartSLOduration=1.246526865 podStartE2EDuration="1.246526865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:07.245743821 +0000 UTC m=+12.277642761" watchObservedRunningTime="2026-10-30 13:49:07.246526865 +0000 UTC m=+12.278425801"
Oct 30 13:49:08.169895 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:08.169864    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nr98w"
Oct 30 13:49:08.209591 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:08.209558    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4" event={"ID":"5c1b1ee0-e5fc-4ce7-b3ad-d9880a5db41f","Type":"ContainerStarted","Data":"95115359ab15bddaafd6485c33256f8ea5cc6e66d4828332615d7d312bcec211"}
Oct 30 13:49:08.209957 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:08.209934    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4"
Oct 30 13:49:08.215798 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:08.215766    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4"
Oct 30 13:49:08.262134 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:08.262048    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zq9x4" podStartSLOduration=1.292424156 podStartE2EDuration="2.262031275s" totalImagesPullingTime="969.607119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:08.261659288 +0000 UTC m=+13.293558227" watchObservedRunningTime="2026-10-30 13:49:08.262031275 +0000 UTC m=+13.293930219"
Oct 30 13:49:08.696779 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:08.696751    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sctxm"
Oct 30 13:49:11.370925 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:49:11.370878    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:49:13.974727 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:13.974689    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"]
Oct 30 13:49:13.991390 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:13.991023    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:13.998884 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:13.998860    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"]
Oct 30 13:49:14.015335 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.015208    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 13:49:14.015452 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.015391    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 13:49:14.028673 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.028647    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 13:49:14.028789 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.028680    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gpsxb\""
Oct 30 13:49:14.028789 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.028739    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 13:49:14.028898 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.028810    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 13:49:14.030362 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.030341    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-28t2s"]
Oct 30 13:49:14.032699 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.032676    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.103829 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.103808    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 13:49:14.104395 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.104362    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hdbtp\""
Oct 30 13:49:14.104512 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.104480    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 13:49:14.115665 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.115643    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 13:49:14.168974 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.168948    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97ed0e87-c8d4-4fab-aa32-8c1da29fce5b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wrgg2\" (UID: \"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.169142 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.168999    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f4f2337-c31d-44cb-b97f-707b5d5d335f-metrics-client-ca\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.169142 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.169080    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-accelerators-collector-config\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.169142 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.169117    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f4f2337-c31d-44cb-b97f-707b5d5d335f-sys\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.169307 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.169174    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f4f2337-c31d-44cb-b97f-707b5d5d335f-root\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.169307 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.169272    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-textfile\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.169378 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.169318    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbpv\" (UniqueName: \"kubernetes.io/projected/2f4f2337-c31d-44cb-b97f-707b5d5d335f-kube-api-access-cnbpv\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.169416 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.169374    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rtbq\" (UniqueName: \"kubernetes.io/projected/97ed0e87-c8d4-4fab-aa32-8c1da29fce5b-kube-api-access-2rtbq\") pod \"openshift-state-metrics-7bd7489957-wrgg2\" (UID: \"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.169457 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.169413    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.169457 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.169442    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97ed0e87-c8d4-4fab-aa32-8c1da29fce5b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wrgg2\" (UID: \"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.169529 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.169506    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-wtmp\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.169587 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.169558    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-tls\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.169637 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.169612    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/97ed0e87-c8d4-4fab-aa32-8c1da29fce5b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wrgg2\" (UID: \"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.270528 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270463    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f4f2337-c31d-44cb-b97f-707b5d5d335f-metrics-client-ca\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.270528 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270503    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-accelerators-collector-config\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.270528 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270528    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f4f2337-c31d-44cb-b97f-707b5d5d335f-sys\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.270797 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270605    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f4f2337-c31d-44cb-b97f-707b5d5d335f-sys\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.270797 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270639    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f4f2337-c31d-44cb-b97f-707b5d5d335f-root\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.270797 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270683    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-textfile\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.270797 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270713    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbpv\" (UniqueName: \"kubernetes.io/projected/2f4f2337-c31d-44cb-b97f-707b5d5d335f-kube-api-access-cnbpv\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.270797 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270720    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f4f2337-c31d-44cb-b97f-707b5d5d335f-root\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.270797 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270740    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rtbq\" (UniqueName: \"kubernetes.io/projected/97ed0e87-c8d4-4fab-aa32-8c1da29fce5b-kube-api-access-2rtbq\") pod \"openshift-state-metrics-7bd7489957-wrgg2\" (UID: \"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.270797 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270779    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.271128 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270819    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97ed0e87-c8d4-4fab-aa32-8c1da29fce5b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wrgg2\" (UID: \"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.271128 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270856    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-wtmp\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.271128 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270884    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-tls\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.271128 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270910    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/97ed0e87-c8d4-4fab-aa32-8c1da29fce5b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wrgg2\" (UID: \"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.271128 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.270974    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97ed0e87-c8d4-4fab-aa32-8c1da29fce5b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wrgg2\" (UID: \"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.271128 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.271091    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-wtmp\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.271128 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.271091    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-textfile\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.271461 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.271190    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-accelerators-collector-config\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.271461 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.271279    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f4f2337-c31d-44cb-b97f-707b5d5d335f-metrics-client-ca\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.271756 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.271731    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97ed0e87-c8d4-4fab-aa32-8c1da29fce5b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wrgg2\" (UID: \"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.273444 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.273417    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-tls\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.273617 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.273597    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/97ed0e87-c8d4-4fab-aa32-8c1da29fce5b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wrgg2\" (UID: \"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.273880 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.273861    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f4f2337-c31d-44cb-b97f-707b5d5d335f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.273948 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.273882    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97ed0e87-c8d4-4fab-aa32-8c1da29fce5b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wrgg2\" (UID: \"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.298458 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.298438    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rtbq\" (UniqueName: \"kubernetes.io/projected/97ed0e87-c8d4-4fab-aa32-8c1da29fce5b-kube-api-access-2rtbq\") pod \"openshift-state-metrics-7bd7489957-wrgg2\" (UID: \"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.298582 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.298456    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbpv\" (UniqueName: \"kubernetes.io/projected/2f4f2337-c31d-44cb-b97f-707b5d5d335f-kube-api-access-cnbpv\") pod \"node-exporter-28t2s\" (UID: \"2f4f2337-c31d-44cb-b97f-707b5d5d335f\") " pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.308224 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.308207    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"
Oct 30 13:49:14.347237 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.347204    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-28t2s"
Oct 30 13:49:14.367099 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:49:14.365663    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4f2337_c31d_44cb_b97f_707b5d5d335f.slice/crio-0c61b2ce0b5b74243b2df17b3a457bd17d737b07742c80a6c76a18d7b6bf417a WatchSource:0}: Error finding container 0c61b2ce0b5b74243b2df17b3a457bd17d737b07742c80a6c76a18d7b6bf417a: Status 404 returned error can't find the container with id 0c61b2ce0b5b74243b2df17b3a457bd17d737b07742c80a6c76a18d7b6bf417a
Oct 30 13:49:14.623901 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:14.623874    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2"]
Oct 30 13:49:14.625424 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:49:14.625378    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ed0e87_c8d4_4fab_aa32_8c1da29fce5b.slice/crio-b7b1b2d773defbd86f88c1a545ffe35ca78dd5ac5f9a41bb1461905f8614f229 WatchSource:0}: Error finding container b7b1b2d773defbd86f88c1a545ffe35ca78dd5ac5f9a41bb1461905f8614f229: Status 404 returned error can't find the container with id b7b1b2d773defbd86f88c1a545ffe35ca78dd5ac5f9a41bb1461905f8614f229
Oct 30 13:49:15.238453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:15.238409    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2" event={"ID":"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b","Type":"ContainerStarted","Data":"feae8b412171d72619f27f7afa402a90617e1b84c1863de1c6c98c6753bb9d50"}
Oct 30 13:49:15.238981 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:15.238479    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2" event={"ID":"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b","Type":"ContainerStarted","Data":"773e52f854017317435ff0361cf402848d574130f87046dc836e9ce17229637e"}
Oct 30 13:49:15.238981 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:15.238510    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2" event={"ID":"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b","Type":"ContainerStarted","Data":"b7b1b2d773defbd86f88c1a545ffe35ca78dd5ac5f9a41bb1461905f8614f229"}
Oct 30 13:49:15.240064 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:15.240035    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28t2s" event={"ID":"2f4f2337-c31d-44cb-b97f-707b5d5d335f","Type":"ContainerStarted","Data":"0c61b2ce0b5b74243b2df17b3a457bd17d737b07742c80a6c76a18d7b6bf417a"}
Oct 30 13:49:16.245441 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:16.245399    4530 generic.go:356] "Generic (PLEG): container finished" podID="2f4f2337-c31d-44cb-b97f-707b5d5d335f" containerID="568e5728e668b58ee86c75f71473cf6f903a6fc23245c8251e4eed6af2e10e5e" exitCode=0
Oct 30 13:49:16.245911 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:16.245582    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28t2s" event={"ID":"2f4f2337-c31d-44cb-b97f-707b5d5d335f","Type":"ContainerDied","Data":"568e5728e668b58ee86c75f71473cf6f903a6fc23245c8251e4eed6af2e10e5e"}
Oct 30 13:49:16.248649 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:16.248615    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2" event={"ID":"97ed0e87-c8d4-4fab-aa32-8c1da29fce5b","Type":"ContainerStarted","Data":"31ce58db72db06d61f69829fe3263ed480edce3f037b8d9964bbb0a75e3692d6"}
Oct 30 13:49:16.353160 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:16.353107    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wrgg2" podStartSLOduration=2.369514172 podStartE2EDuration="3.353087037s" totalImagesPullingTime="983.572865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:16.35264598 +0000 UTC m=+21.384544920" watchObservedRunningTime="2026-10-30 13:49:16.353087037 +0000 UTC m=+21.384985977"
Oct 30 13:49:17.256187 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:17.256072    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28t2s" event={"ID":"2f4f2337-c31d-44cb-b97f-707b5d5d335f","Type":"ContainerStarted","Data":"f904cbebffbfbea26e091b28d0df51774dd086fbb7eca41faecc3c9c370b3ace"}
Oct 30 13:49:17.256187 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:17.256132    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28t2s" event={"ID":"2f4f2337-c31d-44cb-b97f-707b5d5d335f","Type":"ContainerStarted","Data":"af8c3d1174a03fa32be23c610236ed219fa6d8c77b47f51097e1cf4112c53d34"}
Oct 30 13:49:17.333993 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:17.333943    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-28t2s" podStartSLOduration=3.245414469 podStartE2EDuration="4.333926505s" totalImagesPullingTime="1.008664432s" totalInitContainerRuntime="79.847604ms" isStatefulPod=true podCreationTimestamp="2026-10-30 13:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:17.319420522 +0000 UTC m=+22.351319460" watchObservedRunningTime="2026-10-30 13:49:17.333926505 +0000 UTC m=+22.365825446"
Oct 30 13:49:18.809842 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:18.809806    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"]
Oct 30 13:49:18.812376 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:18.812349    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"
Oct 30 13:49:18.831491 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:18.831464    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"]
Oct 30 13:49:18.844316 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:18.844293    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 13:49:18.844446 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:18.844389    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rvx7n\""
Oct 30 13:49:18.909877 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:18.909835    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe7de227-b792-4f2d-9328-b73420af695c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-8kr8n\" (UID: \"fe7de227-b792-4f2d-9328-b73420af695c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"
Oct 30 13:49:18.909877 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:18.909879    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlgx\" (UniqueName: \"kubernetes.io/projected/fe7de227-b792-4f2d-9328-b73420af695c-kube-api-access-ddlgx\") pod \"monitoring-plugin-77f4f76594-8kr8n\" (UID: \"fe7de227-b792-4f2d-9328-b73420af695c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"
Oct 30 13:49:19.010836 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.010798    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe7de227-b792-4f2d-9328-b73420af695c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-8kr8n\" (UID: \"fe7de227-b792-4f2d-9328-b73420af695c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"
Oct 30 13:49:19.010988 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.010841    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlgx\" (UniqueName: \"kubernetes.io/projected/fe7de227-b792-4f2d-9328-b73420af695c-kube-api-access-ddlgx\") pod \"monitoring-plugin-77f4f76594-8kr8n\" (UID: \"fe7de227-b792-4f2d-9328-b73420af695c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"
Oct 30 13:49:19.013618 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.013594    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe7de227-b792-4f2d-9328-b73420af695c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-8kr8n\" (UID: \"fe7de227-b792-4f2d-9328-b73420af695c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"
Oct 30 13:49:19.041242 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.041195    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlgx\" (UniqueName: \"kubernetes.io/projected/fe7de227-b792-4f2d-9328-b73420af695c-kube-api-access-ddlgx\") pod \"monitoring-plugin-77f4f76594-8kr8n\" (UID: \"fe7de227-b792-4f2d-9328-b73420af695c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"
Oct 30 13:49:19.124137 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.124062    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"
Oct 30 13:49:19.428488 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.428410    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"]
Oct 30 13:49:19.432295 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:49:19.432258    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7de227_b792_4f2d_9328_b73420af695c.slice/crio-9ba38c39603cb0c840ed564f5e3035ea3a12920da07c7cdcfa18202e5024eaed WatchSource:0}: Error finding container 9ba38c39603cb0c840ed564f5e3035ea3a12920da07c7cdcfa18202e5024eaed: Status 404 returned error can't find the container with id 9ba38c39603cb0c840ed564f5e3035ea3a12920da07c7cdcfa18202e5024eaed
Oct 30 13:49:19.693854 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.693774    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58555bb787-fjgrq"]
Oct 30 13:49:19.725143 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.725110    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58555bb787-fjgrq"]
Oct 30 13:49:19.725303 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.725245    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.757805 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.757661    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 13:49:19.757805 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.757679    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 13:49:19.757805 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.757696    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 13:49:19.757805 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.757718    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 13:49:19.757805 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.757758    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nh789\""
Oct 30 13:49:19.757805 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.757778    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 13:49:19.787141 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.787118    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 13:49:19.818358 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.818324    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-console-config\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.818774 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.818370    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwmhr\" (UniqueName: \"kubernetes.io/projected/a7380240-664d-4da5-ba21-e9820637890a-kube-api-access-nwmhr\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.818774 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.818450    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-trusted-ca-bundle\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.818774 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.818511    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7380240-664d-4da5-ba21-e9820637890a-tmp\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.818774 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.818556    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-service-ca\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.818774 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.818586    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-oauth-config\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.818774 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.818618    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-serving-cert\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.818774 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.818733    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-oauth-serving-cert\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.919274 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.919244    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7380240-664d-4da5-ba21-e9820637890a-tmp\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.919431 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.919287    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-service-ca\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.919431 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.919331    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-oauth-config\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.919579 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.919528    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-serving-cert\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.919715 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.919609    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-oauth-serving-cert\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.919715 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.919651    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-console-config\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.919715 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.919694    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwmhr\" (UniqueName: \"kubernetes.io/projected/a7380240-664d-4da5-ba21-e9820637890a-kube-api-access-nwmhr\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.919874 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.919730    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-trusted-ca-bundle\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.919874 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.919746    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7380240-664d-4da5-ba21-e9820637890a-tmp\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.920608 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.920077    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-service-ca\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.920608 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.920427    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-oauth-serving-cert\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.920608 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.920565    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-console-config\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.921473 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.921448    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-trusted-ca-bundle\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.922240 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.922221    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-serving-cert\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.922320 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.922275    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-oauth-config\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:19.946733 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:19.946686    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwmhr\" (UniqueName: \"kubernetes.io/projected/a7380240-664d-4da5-ba21-e9820637890a-kube-api-access-nwmhr\") pod \"console-58555bb787-fjgrq\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") " pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:20.049048 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:20.049014    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:20.268161 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:20.268025    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n" event={"ID":"fe7de227-b792-4f2d-9328-b73420af695c","Type":"ContainerStarted","Data":"9ba38c39603cb0c840ed564f5e3035ea3a12920da07c7cdcfa18202e5024eaed"}
Oct 30 13:49:20.373833 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:20.373799    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58555bb787-fjgrq"]
Oct 30 13:49:20.375304 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:49:20.375281    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7380240_664d_4da5_ba21_e9820637890a.slice/crio-51e4d7d83676211d6328808428fc9780ffc5fc07aac540b5080f701bc1dfbb54 WatchSource:0}: Error finding container 51e4d7d83676211d6328808428fc9780ffc5fc07aac540b5080f701bc1dfbb54: Status 404 returned error can't find the container with id 51e4d7d83676211d6328808428fc9780ffc5fc07aac540b5080f701bc1dfbb54
Oct 30 13:49:21.274557 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:21.274513    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58555bb787-fjgrq" event={"ID":"a7380240-664d-4da5-ba21-e9820637890a","Type":"ContainerStarted","Data":"51e4d7d83676211d6328808428fc9780ffc5fc07aac540b5080f701bc1dfbb54"}
Oct 30 13:49:21.953686 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:21.953658    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgvgs"
Oct 30 13:49:22.280901 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:22.280818    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n" event={"ID":"fe7de227-b792-4f2d-9328-b73420af695c","Type":"ContainerStarted","Data":"09a3065f7f3af91ba8cc4235bdf0aa1c31d601b9d6ecfd2af363fc99cc0b193e"}
Oct 30 13:49:22.283617 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:22.283481    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"
Oct 30 13:49:22.288304 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:22.288281    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n"
Oct 30 13:49:22.326618 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:22.326564    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8kr8n" podStartSLOduration=2.440834943 podStartE2EDuration="4.326534549s" totalImagesPullingTime="1.885699606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:22.326226958 +0000 UTC m=+27.358125899" watchObservedRunningTime="2026-10-30 13:49:22.326534549 +0000 UTC m=+27.358433487"
Oct 30 13:49:24.290828 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:24.290789    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-cls76" event={"ID":"12678c60-0b95-404d-aa2d-a10c0ccf91f4","Type":"ContainerStarted","Data":"bd957b68e55deaea2a5d1a5b34f5d0779d53b37da1209a6ad68d83a9b4225df2"}
Oct 30 13:49:24.291278 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:24.290964    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-cls76"
Oct 30 13:49:24.294307 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:24.294283    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58555bb787-fjgrq" event={"ID":"a7380240-664d-4da5-ba21-e9820637890a","Type":"ContainerStarted","Data":"ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc"}
Oct 30 13:49:24.308148 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:24.308123    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-cls76"
Oct 30 13:49:24.354200 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:24.354145    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-cls76" podStartSLOduration=1.898047939 podStartE2EDuration="18.354128873s" totalImagesPullingTime="16.456080934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:24.352816589 +0000 UTC m=+29.384715531" watchObservedRunningTime="2026-10-30 13:49:24.354128873 +0000 UTC m=+29.386027806"
Oct 30 13:49:24.408719 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:24.408666    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58555bb787-fjgrq" podStartSLOduration=2.054283324 podStartE2EDuration="5.408650747s" totalImagesPullingTime="3.354367423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:24.407616721 +0000 UTC m=+29.439515662" watchObservedRunningTime="2026-10-30 13:49:24.408650747 +0000 UTC m=+29.440549685"
Oct 30 13:49:24.726858 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:24.726825    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-98cgg"
Oct 30 13:49:28.215291 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:28.215261    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fb64c5687-zhdxl"
Oct 30 13:49:30.049533 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:30.049499    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:30.050033 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:30.049582    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:30.055020 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:30.054994    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:30.321252 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:30.321180    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:49:41.365397 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:49:41.365319    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:49:49.654125 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.654091    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"]
Oct 30 13:49:49.659829 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.659813    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.678229 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.678196    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"]
Oct 30 13:49:49.691299 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.691273    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 13:49:49.691417 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.691327    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 13:49:49.691417 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.691352    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 13:49:49.691417 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.691357    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 13:49:49.691417 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.691398    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-gm8sr\""
Oct 30 13:49:49.736805 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.736781    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/44b72bcd-b4bc-4a51-9978-d921191ef537-federate-client-tls\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.736913 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.736814    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cszt\" (UniqueName: \"kubernetes.io/projected/44b72bcd-b4bc-4a51-9978-d921191ef537-kube-api-access-4cszt\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.736913 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.736831    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44b72bcd-b4bc-4a51-9978-d921191ef537-metrics-client-ca\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.736999 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.736955    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b72bcd-b4bc-4a51-9978-d921191ef537-serving-certs-ca-bundle\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.736999 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.736982    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44b72bcd-b4bc-4a51-9978-d921191ef537-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.737060 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.737014    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b72bcd-b4bc-4a51-9978-d921191ef537-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.737116 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.737099    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/44b72bcd-b4bc-4a51-9978-d921191ef537-secret-telemeter-client\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.737152 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.737128    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/44b72bcd-b4bc-4a51-9978-d921191ef537-telemeter-client-tls\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.752755 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.752738    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 13:49:49.767729 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.767711    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 13:49:49.838061 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.838032    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/44b72bcd-b4bc-4a51-9978-d921191ef537-federate-client-tls\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.838137 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.838073    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cszt\" (UniqueName: \"kubernetes.io/projected/44b72bcd-b4bc-4a51-9978-d921191ef537-kube-api-access-4cszt\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.838137 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.838094    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44b72bcd-b4bc-4a51-9978-d921191ef537-metrics-client-ca\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.838237 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.838218    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b72bcd-b4bc-4a51-9978-d921191ef537-serving-certs-ca-bundle\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.838282 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.838254    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44b72bcd-b4bc-4a51-9978-d921191ef537-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.838336 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.838293    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b72bcd-b4bc-4a51-9978-d921191ef537-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.838336 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.838325    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/44b72bcd-b4bc-4a51-9978-d921191ef537-secret-telemeter-client\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.838436 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.838358    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/44b72bcd-b4bc-4a51-9978-d921191ef537-telemeter-client-tls\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.838898 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.838874    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44b72bcd-b4bc-4a51-9978-d921191ef537-metrics-client-ca\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.839070 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.839041    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b72bcd-b4bc-4a51-9978-d921191ef537-serving-certs-ca-bundle\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.839189 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.839177    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44b72bcd-b4bc-4a51-9978-d921191ef537-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.840631 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.840611    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/44b72bcd-b4bc-4a51-9978-d921191ef537-secret-telemeter-client\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.840850 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.840829    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/44b72bcd-b4bc-4a51-9978-d921191ef537-telemeter-client-tls\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.840941 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.840926    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/44b72bcd-b4bc-4a51-9978-d921191ef537-federate-client-tls\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.840974 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.840940    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44b72bcd-b4bc-4a51-9978-d921191ef537-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.864751 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.864731    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cszt\" (UniqueName: \"kubernetes.io/projected/44b72bcd-b4bc-4a51-9978-d921191ef537-kube-api-access-4cszt\") pod \"telemeter-client-7bb946c6c-pqnnx\" (UID: \"44b72bcd-b4bc-4a51-9978-d921191ef537\") " pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:49.971902 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:49.971856    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"
Oct 30 13:49:50.297723 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:50.297695    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bb946c6c-pqnnx"]
Oct 30 13:49:50.298357 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:49:50.298333    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b72bcd_b4bc_4a51_9978_d921191ef537.slice/crio-3ed0e1fcbdc88f9012813f6f37b1bc0fe157a286314fa1fd78d174e46a51e821 WatchSource:0}: Error finding container 3ed0e1fcbdc88f9012813f6f37b1bc0fe157a286314fa1fd78d174e46a51e821: Status 404 returned error can't find the container with id 3ed0e1fcbdc88f9012813f6f37b1bc0fe157a286314fa1fd78d174e46a51e821
Oct 30 13:49:50.386320 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:50.386294    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx" event={"ID":"44b72bcd-b4bc-4a51-9978-d921191ef537","Type":"ContainerStarted","Data":"3ed0e1fcbdc88f9012813f6f37b1bc0fe157a286314fa1fd78d174e46a51e821"}
Oct 30 13:49:52.394767 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:52.394731    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx" event={"ID":"44b72bcd-b4bc-4a51-9978-d921191ef537","Type":"ContainerStarted","Data":"dfd9eb6670ed92128e710247f6198c55a969f49ee3be14a316b829ac87f80a6a"}
Oct 30 13:49:54.403078 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.403039    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx" event={"ID":"44b72bcd-b4bc-4a51-9978-d921191ef537","Type":"ContainerStarted","Data":"962c00d768458e8aa5238b8e98ee4d06498b8505594196ba15c4692ecfa051b9"}
Oct 30 13:49:54.403536 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.403107    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx" event={"ID":"44b72bcd-b4bc-4a51-9978-d921191ef537","Type":"ContainerStarted","Data":"aa13c96bb3d2f15e270a31ea219d7e23b4849c2bfc2cba01f7214e5a5b057c9e"}
Oct 30 13:49:54.484584 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.484523    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7bb946c6c-pqnnx" podStartSLOduration=2.24027954 podStartE2EDuration="5.484511146s" totalImagesPullingTime="3.244231606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:54.469921759 +0000 UTC m=+59.501820699" watchObservedRunningTime="2026-10-30 13:49:54.484511146 +0000 UTC m=+59.516410098"
Oct 30 13:49:54.661054 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.660999    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bb6c8cb9b-8v75t"]
Oct 30 13:49:54.684645 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.684626    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.688236 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.688213    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb6c8cb9b-8v75t"]
Oct 30 13:49:54.772836 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.772813    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-trusted-ca-bundle\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.772966 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.772854    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-service-ca\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.772966 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.772881    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-oauth-serving-cert\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.773053 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.772981    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-console-config\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.773053 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.773032    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c011057-9cb5-4c50-904e-6e5c70d74637-tmp\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.773144 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.773053    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-serving-cert\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.773144 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.773125    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-oauth-config\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.773253 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.773180    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45ggg\" (UniqueName: \"kubernetes.io/projected/9c011057-9cb5-4c50-904e-6e5c70d74637-kube-api-access-45ggg\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.874145 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.874113    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c011057-9cb5-4c50-904e-6e5c70d74637-tmp\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.874262 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.874189    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-serving-cert\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.874262 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.874216    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-oauth-config\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.874262 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.874231    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45ggg\" (UniqueName: \"kubernetes.io/projected/9c011057-9cb5-4c50-904e-6e5c70d74637-kube-api-access-45ggg\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.874262 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.874256    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-trusted-ca-bundle\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.874454 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.874277    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-service-ca\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.874454 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.874302    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-oauth-serving-cert\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.874454 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.874344    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-console-config\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.874612 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.874561    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c011057-9cb5-4c50-904e-6e5c70d74637-tmp\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.875021 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.875000    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-service-ca\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.875146 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.875124    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-oauth-serving-cert\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.875211 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.875152    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-trusted-ca-bundle\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.875211 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.875168    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-console-config\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.876565 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.876530    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-serving-cert\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.876732 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.876715    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-oauth-config\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:54.901805 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:54.901780    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45ggg\" (UniqueName: \"kubernetes.io/projected/9c011057-9cb5-4c50-904e-6e5c70d74637-kube-api-access-45ggg\") pod \"console-7bb6c8cb9b-8v75t\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") " pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:55.009962 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.009905    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:49:55.342952 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.342924    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb6c8cb9b-8v75t"]
Oct 30 13:49:55.343846 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:49:55.343815    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c011057_9cb5_4c50_904e_6e5c70d74637.slice/crio-6acf965721948419231f6a47c5a8303df97979a0a2ae539ba8f90562a2a5ad15 WatchSource:0}: Error finding container 6acf965721948419231f6a47c5a8303df97979a0a2ae539ba8f90562a2a5ad15: Status 404 returned error can't find the container with id 6acf965721948419231f6a47c5a8303df97979a0a2ae539ba8f90562a2a5ad15
Oct 30 13:49:55.407787 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.407755    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb6c8cb9b-8v75t" event={"ID":"9c011057-9cb5-4c50-904e-6e5c70d74637","Type":"ContainerStarted","Data":"6acf965721948419231f6a47c5a8303df97979a0a2ae539ba8f90562a2a5ad15"}
Oct 30 13:49:55.457843 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.457650    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-f4lw9-sg475"]
Oct 30 13:49:55.458111 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.458085    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-f4lw9-sg475" podUID="c6f44206-dd53-4dae-b68c-84a8317227b5" containerName="insights-gathering" containerID="cri-o://07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258" gracePeriod=30
Oct 30 13:49:55.653499 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.653470    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-s6pdm-v2hc4"]
Oct 30 13:49:55.674941 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.674914    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-s6pdm-v2hc4"]
Oct 30 13:49:55.675068 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.675039    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.729284 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.729262    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:49:55.780030 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.779999    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-service-ca-bundle\") pod \"periodic-gathering-s6pdm-v2hc4\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") " pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.780158 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.780044    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-trusted-ca-bundle\") pod \"periodic-gathering-s6pdm-v2hc4\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") " pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.780158 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.780062    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bssr9\" (UniqueName: \"kubernetes.io/projected/051417fc-81ae-4c7e-b30d-6b0e751a6c09-kube-api-access-bssr9\") pod \"periodic-gathering-s6pdm-v2hc4\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") " pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.780288 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.780194    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/051417fc-81ae-4c7e-b30d-6b0e751a6c09-archives-path\") pod \"periodic-gathering-s6pdm-v2hc4\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") " pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.881237 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881182    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f44206-dd53-4dae-b68c-84a8317227b5-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6f44206-dd53-4dae-b68c-84a8317227b5-archives-path\") pod \"c6f44206-dd53-4dae-b68c-84a8317227b5\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") "
Oct 30 13:49:55.881237 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881223    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f44206-dd53-4dae-b68c-84a8317227b5-kube-api-access-swft4\" (UniqueName: \"kubernetes.io/projected/c6f44206-dd53-4dae-b68c-84a8317227b5-kube-api-access-swft4\") pod \"c6f44206-dd53-4dae-b68c-84a8317227b5\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") "
Oct 30 13:49:55.881375 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881261    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-service-ca-bundle\") pod \"c6f44206-dd53-4dae-b68c-84a8317227b5\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") "
Oct 30 13:49:55.881375 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881315    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-trusted-ca-bundle\") pod \"c6f44206-dd53-4dae-b68c-84a8317227b5\" (UID: \"c6f44206-dd53-4dae-b68c-84a8317227b5\") "
Oct 30 13:49:55.881773 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881429    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-service-ca-bundle\") pod \"periodic-gathering-s6pdm-v2hc4\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") " pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.881773 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881487    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-trusted-ca-bundle\") pod \"periodic-gathering-s6pdm-v2hc4\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") " pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.881773 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881517    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bssr9\" (UniqueName: \"kubernetes.io/projected/051417fc-81ae-4c7e-b30d-6b0e751a6c09-kube-api-access-bssr9\") pod \"periodic-gathering-s6pdm-v2hc4\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") " pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.881773 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881488    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f44206-dd53-4dae-b68c-84a8317227b5-archives-path" pod "c6f44206-dd53-4dae-b68c-84a8317227b5" (UID: "c6f44206-dd53-4dae-b68c-84a8317227b5"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:49:55.881773 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881587    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/051417fc-81ae-4c7e-b30d-6b0e751a6c09-archives-path\") pod \"periodic-gathering-s6pdm-v2hc4\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") " pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.881773 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881665    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-service-ca-bundle" pod "c6f44206-dd53-4dae-b68c-84a8317227b5" (UID: "c6f44206-dd53-4dae-b68c-84a8317227b5"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:55.881773 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881677    4530 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6f44206-dd53-4dae-b68c-84a8317227b5-archives-path\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:49:55.882252 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881806    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-trusted-ca-bundle" pod "c6f44206-dd53-4dae-b68c-84a8317227b5" (UID: "c6f44206-dd53-4dae-b68c-84a8317227b5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:55.882252 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.881978    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/051417fc-81ae-4c7e-b30d-6b0e751a6c09-archives-path\") pod \"periodic-gathering-s6pdm-v2hc4\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") " pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.882252 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.882216    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-service-ca-bundle\") pod \"periodic-gathering-s6pdm-v2hc4\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") " pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.882569 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.882447    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-trusted-ca-bundle\") pod \"periodic-gathering-s6pdm-v2hc4\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") " pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.883929 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.883908    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f44206-dd53-4dae-b68c-84a8317227b5-kube-api-access-swft4" pod "c6f44206-dd53-4dae-b68c-84a8317227b5" (UID: "c6f44206-dd53-4dae-b68c-84a8317227b5"). InnerVolumeSpecName "kube-api-access-swft4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:55.909781 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.909756    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bssr9\" (UniqueName: \"kubernetes.io/projected/051417fc-81ae-4c7e-b30d-6b0e751a6c09-kube-api-access-bssr9\") pod \"periodic-gathering-s6pdm-v2hc4\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") " pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:55.982695 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.982675    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swft4\" (UniqueName: \"kubernetes.io/projected/c6f44206-dd53-4dae-b68c-84a8317227b5-kube-api-access-swft4\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:49:55.982695 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.982695    4530 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-service-ca-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:49:55.982799 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:55.982704    4530 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6f44206-dd53-4dae-b68c-84a8317227b5-trusted-ca-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:49:56.007659 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.007639    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:49:56.332815 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.332783    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-s6pdm-v2hc4"]
Oct 30 13:49:56.333164 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:49:56.333142    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051417fc_81ae_4c7e_b30d_6b0e751a6c09.slice/crio-bcb9a2090fb730f2c6c9e1e901a5e02c34af04aa01ba73487234d221722ac6bf WatchSource:0}: Error finding container bcb9a2090fb730f2c6c9e1e901a5e02c34af04aa01ba73487234d221722ac6bf: Status 404 returned error can't find the container with id bcb9a2090fb730f2c6c9e1e901a5e02c34af04aa01ba73487234d221722ac6bf
Oct 30 13:49:56.413009 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.412980    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb6c8cb9b-8v75t" event={"ID":"9c011057-9cb5-4c50-904e-6e5c70d74637","Type":"ContainerStarted","Data":"f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63"}
Oct 30 13:49:56.414381 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.414351    4530 generic.go:356] "Generic (PLEG): container finished" podID="c6f44206-dd53-4dae-b68c-84a8317227b5" containerID="07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258" exitCode=2
Oct 30 13:49:56.414469 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.414387    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f4lw9-sg475"
Oct 30 13:49:56.414469 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.414427    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4lw9-sg475" event={"ID":"c6f44206-dd53-4dae-b68c-84a8317227b5","Type":"ContainerDied","Data":"07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258"}
Oct 30 13:49:56.414579 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.414469    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f4lw9-sg475" event={"ID":"c6f44206-dd53-4dae-b68c-84a8317227b5","Type":"ContainerDied","Data":"7e7b183942858b307e989212f04f7088a104787ee77e63ff40852eb888bc702d"}
Oct 30 13:49:56.414579 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.414497    4530 scope.go:122] "RemoveContainer" containerID="07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258"
Oct 30 13:49:56.416230 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.416201    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s6pdm-v2hc4" event={"ID":"051417fc-81ae-4c7e-b30d-6b0e751a6c09","Type":"ContainerStarted","Data":"dec37670f97fa18fef0e0513ca0945c0a7cc31bc5a161b86056430d73ce2d6fb"}
Oct 30 13:49:56.416312 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.416255    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s6pdm-v2hc4" event={"ID":"051417fc-81ae-4c7e-b30d-6b0e751a6c09","Type":"ContainerStarted","Data":"bcb9a2090fb730f2c6c9e1e901a5e02c34af04aa01ba73487234d221722ac6bf"}
Oct 30 13:49:56.421878 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.421856    4530 scope.go:122] "RemoveContainer" containerID="07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258"
Oct 30 13:49:56.422210 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:49:56.422181    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258\": container with ID starting with 07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258 not found: ID does not exist" containerID="07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258"
Oct 30 13:49:56.422301 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.422223    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258"} err="failed to get container status \"07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258\": rpc error: code = NotFound desc = could not find container \"07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258\": container with ID starting with 07d83ae6903286b1c993672935e4b07999f3dfe79c683793525047caa52e9258 not found: ID does not exist"
Oct 30 13:49:56.466331 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.466272    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7bb6c8cb9b-8v75t" podStartSLOduration=2.466258984 podStartE2EDuration="2.466258984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:56.464870306 +0000 UTC m=+61.496769245" watchObservedRunningTime="2026-10-30 13:49:56.466258984 +0000 UTC m=+61.498157922"
Oct 30 13:49:56.506364 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.506339    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-f4lw9-sg475"]
Oct 30 13:49:56.508752 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:56.508732    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-f4lw9-sg475"]
Oct 30 13:49:57.112035 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:57.112002    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f44206-dd53-4dae-b68c-84a8317227b5" path="/var/lib/kubelet/pods/c6f44206-dd53-4dae-b68c-84a8317227b5/volumes"
Oct 30 13:49:57.467815 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:49:57.467717    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-s6pdm-v2hc4" podStartSLOduration=2.467703248 podStartE2EDuration="2.467703248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:57.466889199 +0000 UTC m=+62.498788137" watchObservedRunningTime="2026-10-30 13:49:57.467703248 +0000 UTC m=+62.499602187"
Oct 30 13:50:00.359477 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:00.359419    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xpz2s_6bd2311e-929b-4d5a-9857-47c0500c3be1/serve-healthcheck-canary/0.log"
Oct 30 13:50:05.010374 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:05.010313    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:50:05.010374 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:05.010373    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:50:05.014910 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:05.014890    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:50:05.450338 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:05.450309    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:50:05.533159 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:05.533129    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58555bb787-fjgrq"]
Oct 30 13:50:11.365274 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:50:11.365240    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:50:30.558326 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.558259    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58555bb787-fjgrq" podUID="a7380240-664d-4da5-ba21-e9820637890a" containerName="console" containerID="cri-o://ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc" gracePeriod=15
Oct 30 13:50:30.790215 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.790193    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:50:30.928786 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.928727    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-console-config\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-console-config\") pod \"a7380240-664d-4da5-ba21-e9820637890a\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") "
Oct 30 13:50:30.928906 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.928813    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-trusted-ca-bundle\") pod \"a7380240-664d-4da5-ba21-e9820637890a\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") "
Oct 30 13:50:30.928906 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.928844    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7380240-664d-4da5-ba21-e9820637890a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7380240-664d-4da5-ba21-e9820637890a-tmp\") pod \"a7380240-664d-4da5-ba21-e9820637890a\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") "
Oct 30 13:50:30.928906 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.928874    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-oauth-serving-cert\") pod \"a7380240-664d-4da5-ba21-e9820637890a\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") "
Oct 30 13:50:30.929075 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.928903    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-service-ca\") pod \"a7380240-664d-4da5-ba21-e9820637890a\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") "
Oct 30 13:50:30.929075 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.928938    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7380240-664d-4da5-ba21-e9820637890a-kube-api-access-nwmhr\" (UniqueName: \"kubernetes.io/projected/a7380240-664d-4da5-ba21-e9820637890a-kube-api-access-nwmhr\") pod \"a7380240-664d-4da5-ba21-e9820637890a\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") "
Oct 30 13:50:30.929075 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.929005    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-oauth-config\") pod \"a7380240-664d-4da5-ba21-e9820637890a\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") "
Oct 30 13:50:30.929075 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.929033    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-serving-cert\") pod \"a7380240-664d-4da5-ba21-e9820637890a\" (UID: \"a7380240-664d-4da5-ba21-e9820637890a\") "
Oct 30 13:50:30.929247 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.929133    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7380240-664d-4da5-ba21-e9820637890a-tmp" pod "a7380240-664d-4da5-ba21-e9820637890a" (UID: "a7380240-664d-4da5-ba21-e9820637890a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:50:30.929247 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.929168    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-console-config" pod "a7380240-664d-4da5-ba21-e9820637890a" (UID: "a7380240-664d-4da5-ba21-e9820637890a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:30.929323 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.929299    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-oauth-serving-cert" pod "a7380240-664d-4da5-ba21-e9820637890a" (UID: "a7380240-664d-4da5-ba21-e9820637890a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:30.929323 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.929309    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-trusted-ca-bundle" pod "a7380240-664d-4da5-ba21-e9820637890a" (UID: "a7380240-664d-4da5-ba21-e9820637890a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:30.929392 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.929372    4530 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-trusted-ca-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:50:30.929431 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.929391    4530 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7380240-664d-4da5-ba21-e9820637890a-tmp\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:50:30.929431 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.929391    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-service-ca" pod "a7380240-664d-4da5-ba21-e9820637890a" (UID: "a7380240-664d-4da5-ba21-e9820637890a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:30.929431 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.929407    4530 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-oauth-serving-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:50:30.929431 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.929423    4530 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-console-config\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:50:30.931372 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.931338    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-oauth-config" pod "a7380240-664d-4da5-ba21-e9820637890a" (UID: "a7380240-664d-4da5-ba21-e9820637890a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:30.931372 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.931356    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-serving-cert" pod "a7380240-664d-4da5-ba21-e9820637890a" (UID: "a7380240-664d-4da5-ba21-e9820637890a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:30.931524 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:30.931447    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7380240-664d-4da5-ba21-e9820637890a-kube-api-access-nwmhr" pod "a7380240-664d-4da5-ba21-e9820637890a" (UID: "a7380240-664d-4da5-ba21-e9820637890a"). InnerVolumeSpecName "kube-api-access-nwmhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:31.029961 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.029918    4530 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7380240-664d-4da5-ba21-e9820637890a-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:50:31.029961 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.029960    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwmhr\" (UniqueName: \"kubernetes.io/projected/a7380240-664d-4da5-ba21-e9820637890a-kube-api-access-nwmhr\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:50:31.030130 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.029977    4530 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-oauth-config\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:50:31.030130 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.029990    4530 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7380240-664d-4da5-ba21-e9820637890a-console-serving-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:50:31.532488 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.532459    4530 generic.go:356] "Generic (PLEG): container finished" podID="a7380240-664d-4da5-ba21-e9820637890a" containerID="ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc" exitCode=0
Oct 30 13:50:31.532650 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.532494    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58555bb787-fjgrq" event={"ID":"a7380240-664d-4da5-ba21-e9820637890a","Type":"ContainerDied","Data":"ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc"}
Oct 30 13:50:31.532650 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.532505    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58555bb787-fjgrq"
Oct 30 13:50:31.532650 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.532529    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58555bb787-fjgrq" event={"ID":"a7380240-664d-4da5-ba21-e9820637890a","Type":"ContainerDied","Data":"51e4d7d83676211d6328808428fc9780ffc5fc07aac540b5080f701bc1dfbb54"}
Oct 30 13:50:31.532650 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.532563    4530 scope.go:122] "RemoveContainer" containerID="ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc"
Oct 30 13:50:31.540276 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.540259    4530 scope.go:122] "RemoveContainer" containerID="ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc"
Oct 30 13:50:31.540575 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:50:31.540557    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc\": container with ID starting with ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc not found: ID does not exist" containerID="ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc"
Oct 30 13:50:31.540640 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.540588    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc"} err="failed to get container status \"ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc\": rpc error: code = NotFound desc = could not find container \"ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc\": container with ID starting with ffa70144c169429c258490ba05f65961913fe159769b31e70493ed875707a1fc not found: ID does not exist"
Oct 30 13:50:31.591702 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.591678    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58555bb787-fjgrq"]
Oct 30 13:50:31.605748 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:31.605727    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58555bb787-fjgrq"]
Oct 30 13:50:33.111682 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:50:33.111651    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7380240-664d-4da5-ba21-e9820637890a" path="/var/lib/kubelet/pods/a7380240-664d-4da5-ba21-e9820637890a/volumes"
Oct 30 13:50:41.365045 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:50:41.365008    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:51:03.099693 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.099659    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-79d7dd84cd-p798f"]
Oct 30 13:51:03.100084 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.099915    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c6f44206-dd53-4dae-b68c-84a8317227b5" containerName="insights-gathering"
Oct 30 13:51:03.100084 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.099926    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f44206-dd53-4dae-b68c-84a8317227b5" containerName="insights-gathering"
Oct 30 13:51:03.100084 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.099945    4530 cpu_manager.go:414] "containerMap: removing container" podUID="a7380240-664d-4da5-ba21-e9820637890a" containerName="console"
Oct 30 13:51:03.100084 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.099950    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7380240-664d-4da5-ba21-e9820637890a" containerName="console"
Oct 30 13:51:03.100084 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.099999    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f44206-dd53-4dae-b68c-84a8317227b5" containerName="insights-gathering"
Oct 30 13:51:03.100084 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.100008    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7380240-664d-4da5-ba21-e9820637890a" containerName="console"
Oct 30 13:51:03.103937 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.103913    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.131806 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.131778    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6e4eb76-fb65-4540-941f-1b418427c124-tmp\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.131922 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.131815    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6e4eb76-fb65-4540-941f-1b418427c124-console-oauth-config\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.131922 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.131836    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e4eb76-fb65-4540-941f-1b418427c124-service-ca\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.132042 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.131961    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6e4eb76-fb65-4540-941f-1b418427c124-oauth-serving-cert\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.132042 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.131989    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6e4eb76-fb65-4540-941f-1b418427c124-console-serving-cert\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.132042 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.132014    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6e4eb76-fb65-4540-941f-1b418427c124-console-config\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.132199 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.132093    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e4eb76-fb65-4540-941f-1b418427c124-trusted-ca-bundle\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.132199 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.132114    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh94g\" (UniqueName: \"kubernetes.io/projected/c6e4eb76-fb65-4540-941f-1b418427c124-kube-api-access-xh94g\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.137760 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.137740    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79d7dd84cd-p798f"]
Oct 30 13:51:03.232881 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.232852    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e4eb76-fb65-4540-941f-1b418427c124-trusted-ca-bundle\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.232993 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.232888    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh94g\" (UniqueName: \"kubernetes.io/projected/c6e4eb76-fb65-4540-941f-1b418427c124-kube-api-access-xh94g\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.232993 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.232942    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6e4eb76-fb65-4540-941f-1b418427c124-tmp\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.232993 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.232969    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6e4eb76-fb65-4540-941f-1b418427c124-console-oauth-config\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.232993 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.232990    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e4eb76-fb65-4540-941f-1b418427c124-service-ca\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.233202 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.233035    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6e4eb76-fb65-4540-941f-1b418427c124-oauth-serving-cert\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.233202 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.233061    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6e4eb76-fb65-4540-941f-1b418427c124-console-serving-cert\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.233202 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.233102    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6e4eb76-fb65-4540-941f-1b418427c124-console-config\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.233416 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.233381    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6e4eb76-fb65-4540-941f-1b418427c124-tmp\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.233763 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.233738    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e4eb76-fb65-4540-941f-1b418427c124-trusted-ca-bundle\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.233881 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.233783    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e4eb76-fb65-4540-941f-1b418427c124-service-ca\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.233946 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.233882    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6e4eb76-fb65-4540-941f-1b418427c124-oauth-serving-cert\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.233946 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.233921    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6e4eb76-fb65-4540-941f-1b418427c124-console-config\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.235480 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.235455    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6e4eb76-fb65-4540-941f-1b418427c124-console-oauth-config\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.235581 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.235492    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6e4eb76-fb65-4540-941f-1b418427c124-console-serving-cert\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.259960 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.259937    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh94g\" (UniqueName: \"kubernetes.io/projected/c6e4eb76-fb65-4540-941f-1b418427c124-kube-api-access-xh94g\") pod \"console-79d7dd84cd-p798f\" (UID: \"c6e4eb76-fb65-4540-941f-1b418427c124\") " pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.415939 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.415868    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:03.725587 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:03.725558    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79d7dd84cd-p798f"]
Oct 30 13:51:03.726020 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:51:03.726001    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e4eb76_fb65_4540_941f_1b418427c124.slice/crio-66b5ac28128b316acdc224fef7e37d37bf884814404957aa1c9572313c262fec WatchSource:0}: Error finding container 66b5ac28128b316acdc224fef7e37d37bf884814404957aa1c9572313c262fec: Status 404 returned error can't find the container with id 66b5ac28128b316acdc224fef7e37d37bf884814404957aa1c9572313c262fec
Oct 30 13:51:04.640528 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:04.640494    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d7dd84cd-p798f" event={"ID":"c6e4eb76-fb65-4540-941f-1b418427c124","Type":"ContainerStarted","Data":"25b350e83127ba4a4765f6c9753d3df2f28419488d96f63c297c21c5af4be5a0"}
Oct 30 13:51:04.640906 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:04.640557    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d7dd84cd-p798f" event={"ID":"c6e4eb76-fb65-4540-941f-1b418427c124","Type":"ContainerStarted","Data":"66b5ac28128b316acdc224fef7e37d37bf884814404957aa1c9572313c262fec"}
Oct 30 13:51:04.688803 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:04.688747    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-79d7dd84cd-p798f" podStartSLOduration=1.6887339369999999 podStartE2EDuration="1.688733937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:04.688086393 +0000 UTC m=+129.719985331" watchObservedRunningTime="2026-10-30 13:51:04.688733937 +0000 UTC m=+129.720632886"
Oct 30 13:51:11.364460 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:51:11.364384    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:51:13.416788 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:13.416743    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:13.417252 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:13.416814    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:13.422056 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:13.422034    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:13.674467 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:13.674395    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79d7dd84cd-p798f"
Oct 30 13:51:13.756552 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:13.756509    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb6c8cb9b-8v75t"]
Oct 30 13:51:21.697369 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:21.697336    4530 generic.go:356] "Generic (PLEG): container finished" podID="051417fc-81ae-4c7e-b30d-6b0e751a6c09" containerID="dec37670f97fa18fef0e0513ca0945c0a7cc31bc5a161b86056430d73ce2d6fb" exitCode=0
Oct 30 13:51:21.697837 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:21.697392    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s6pdm-v2hc4" event={"ID":"051417fc-81ae-4c7e-b30d-6b0e751a6c09","Type":"ContainerDied","Data":"dec37670f97fa18fef0e0513ca0945c0a7cc31bc5a161b86056430d73ce2d6fb"}
Oct 30 13:51:22.827031 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.827005    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:51:22.861750 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.861725    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-service-ca-bundle\") pod \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") "
Oct 30 13:51:22.861822 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.861766    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051417fc-81ae-4c7e-b30d-6b0e751a6c09-kube-api-access-bssr9\" (UniqueName: \"kubernetes.io/projected/051417fc-81ae-4c7e-b30d-6b0e751a6c09-kube-api-access-bssr9\") pod \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") "
Oct 30 13:51:22.861822 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.861792    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051417fc-81ae-4c7e-b30d-6b0e751a6c09-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/051417fc-81ae-4c7e-b30d-6b0e751a6c09-archives-path\") pod \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") "
Oct 30 13:51:22.861912 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.861833    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-trusted-ca-bundle\") pod \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\" (UID: \"051417fc-81ae-4c7e-b30d-6b0e751a6c09\") "
Oct 30 13:51:22.862178 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.862123    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051417fc-81ae-4c7e-b30d-6b0e751a6c09-archives-path" pod "051417fc-81ae-4c7e-b30d-6b0e751a6c09" (UID: "051417fc-81ae-4c7e-b30d-6b0e751a6c09"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:51:22.862240 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.862225    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-service-ca-bundle" pod "051417fc-81ae-4c7e-b30d-6b0e751a6c09" (UID: "051417fc-81ae-4c7e-b30d-6b0e751a6c09"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:22.862275 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.862235    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-trusted-ca-bundle" pod "051417fc-81ae-4c7e-b30d-6b0e751a6c09" (UID: "051417fc-81ae-4c7e-b30d-6b0e751a6c09"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:22.863703 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.863685    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051417fc-81ae-4c7e-b30d-6b0e751a6c09-kube-api-access-bssr9" pod "051417fc-81ae-4c7e-b30d-6b0e751a6c09" (UID: "051417fc-81ae-4c7e-b30d-6b0e751a6c09"). InnerVolumeSpecName "kube-api-access-bssr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:51:22.963240 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.963184    4530 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-service-ca-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:51:22.963240 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.963204    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bssr9\" (UniqueName: \"kubernetes.io/projected/051417fc-81ae-4c7e-b30d-6b0e751a6c09-kube-api-access-bssr9\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:51:22.963240 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.963214    4530 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/051417fc-81ae-4c7e-b30d-6b0e751a6c09-archives-path\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:51:22.963240 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:22.963224    4530 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051417fc-81ae-4c7e-b30d-6b0e751a6c09-trusted-ca-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:51:23.705196 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:23.705168    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s6pdm-v2hc4" event={"ID":"051417fc-81ae-4c7e-b30d-6b0e751a6c09","Type":"ContainerDied","Data":"bcb9a2090fb730f2c6c9e1e901a5e02c34af04aa01ba73487234d221722ac6bf"}
Oct 30 13:51:23.705358 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:23.705212    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcb9a2090fb730f2c6c9e1e901a5e02c34af04aa01ba73487234d221722ac6bf"
Oct 30 13:51:23.705358 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:23.705181    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s6pdm-v2hc4"
Oct 30 13:51:38.781178 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:38.781139    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7bb6c8cb9b-8v75t" podUID="9c011057-9cb5-4c50-904e-6e5c70d74637" containerName="console" containerID="cri-o://f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63" gracePeriod=15
Oct 30 13:51:39.018557 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.018518    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:51:39.066141 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066119    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-oauth-serving-cert\") pod \"9c011057-9cb5-4c50-904e-6e5c70d74637\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") "
Oct 30 13:51:39.066280 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066168    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-trusted-ca-bundle\") pod \"9c011057-9cb5-4c50-904e-6e5c70d74637\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") "
Oct 30 13:51:39.066280 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066198    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-service-ca\") pod \"9c011057-9cb5-4c50-904e-6e5c70d74637\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") "
Oct 30 13:51:39.066449 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066342    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-oauth-config\") pod \"9c011057-9cb5-4c50-904e-6e5c70d74637\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") "
Oct 30 13:51:39.066449 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066397    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-serving-cert\") pod \"9c011057-9cb5-4c50-904e-6e5c70d74637\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") "
Oct 30 13:51:39.066573 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066447    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c011057-9cb5-4c50-904e-6e5c70d74637-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c011057-9cb5-4c50-904e-6e5c70d74637-tmp\") pod \"9c011057-9cb5-4c50-904e-6e5c70d74637\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") "
Oct 30 13:51:39.066573 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066485    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c011057-9cb5-4c50-904e-6e5c70d74637-kube-api-access-45ggg\" (UniqueName: \"kubernetes.io/projected/9c011057-9cb5-4c50-904e-6e5c70d74637-kube-api-access-45ggg\") pod \"9c011057-9cb5-4c50-904e-6e5c70d74637\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") "
Oct 30 13:51:39.066682 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066579    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-console-config\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-console-config\") pod \"9c011057-9cb5-4c50-904e-6e5c70d74637\" (UID: \"9c011057-9cb5-4c50-904e-6e5c70d74637\") "
Oct 30 13:51:39.066682 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066610    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-trusted-ca-bundle" pod "9c011057-9cb5-4c50-904e-6e5c70d74637" (UID: "9c011057-9cb5-4c50-904e-6e5c70d74637"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:39.066682 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066628    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-service-ca" pod "9c011057-9cb5-4c50-904e-6e5c70d74637" (UID: "9c011057-9cb5-4c50-904e-6e5c70d74637"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:39.066682 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066616    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-oauth-serving-cert" pod "9c011057-9cb5-4c50-904e-6e5c70d74637" (UID: "9c011057-9cb5-4c50-904e-6e5c70d74637"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:39.066900 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066879    4530 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:51:39.066958 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066902    4530 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-oauth-serving-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:51:39.066958 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066918    4530 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-trusted-ca-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:51:39.067057 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.066966    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-console-config" pod "9c011057-9cb5-4c50-904e-6e5c70d74637" (UID: "9c011057-9cb5-4c50-904e-6e5c70d74637"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:39.067330 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.067306    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c011057-9cb5-4c50-904e-6e5c70d74637-tmp" pod "9c011057-9cb5-4c50-904e-6e5c70d74637" (UID: "9c011057-9cb5-4c50-904e-6e5c70d74637"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:51:39.069188 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.069159    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c011057-9cb5-4c50-904e-6e5c70d74637-kube-api-access-45ggg" pod "9c011057-9cb5-4c50-904e-6e5c70d74637" (UID: "9c011057-9cb5-4c50-904e-6e5c70d74637"). InnerVolumeSpecName "kube-api-access-45ggg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:51:39.069270 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.069219    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-oauth-config" pod "9c011057-9cb5-4c50-904e-6e5c70d74637" (UID: "9c011057-9cb5-4c50-904e-6e5c70d74637"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:51:39.069270 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.069250    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-serving-cert" pod "9c011057-9cb5-4c50-904e-6e5c70d74637" (UID: "9c011057-9cb5-4c50-904e-6e5c70d74637"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:51:39.168121 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.168099    4530 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-serving-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:51:39.168121 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.168117    4530 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c011057-9cb5-4c50-904e-6e5c70d74637-tmp\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:51:39.168250 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.168127    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45ggg\" (UniqueName: \"kubernetes.io/projected/9c011057-9cb5-4c50-904e-6e5c70d74637-kube-api-access-45ggg\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:51:39.168250 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.168136    4530 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c011057-9cb5-4c50-904e-6e5c70d74637-console-config\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:51:39.168250 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.168144    4530 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c011057-9cb5-4c50-904e-6e5c70d74637-console-oauth-config\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:51:39.758983 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.758946    4530 generic.go:356] "Generic (PLEG): container finished" podID="9c011057-9cb5-4c50-904e-6e5c70d74637" containerID="f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63" exitCode=0
Oct 30 13:51:39.759160 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.759007    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb6c8cb9b-8v75t" event={"ID":"9c011057-9cb5-4c50-904e-6e5c70d74637","Type":"ContainerDied","Data":"f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63"}
Oct 30 13:51:39.759160 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.759013    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb6c8cb9b-8v75t"
Oct 30 13:51:39.759160 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.759051    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb6c8cb9b-8v75t" event={"ID":"9c011057-9cb5-4c50-904e-6e5c70d74637","Type":"ContainerDied","Data":"6acf965721948419231f6a47c5a8303df97979a0a2ae539ba8f90562a2a5ad15"}
Oct 30 13:51:39.759160 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.759072    4530 scope.go:122] "RemoveContainer" containerID="f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63"
Oct 30 13:51:39.766282 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.766260    4530 scope.go:122] "RemoveContainer" containerID="f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63"
Oct 30 13:51:39.766569 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:51:39.766528    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63\": container with ID starting with f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63 not found: ID does not exist" containerID="f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63"
Oct 30 13:51:39.766655 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.766576    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63"} err="failed to get container status \"f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63\": rpc error: code = NotFound desc = could not find container \"f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63\": container with ID starting with f11495d6402f2c906f767c1f9aa91339aed094ae18111f1dfa1fc65f64be5e63 not found: ID does not exist"
Oct 30 13:51:39.818174 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.818147    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb6c8cb9b-8v75t"]
Oct 30 13:51:39.823773 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:39.823750    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bb6c8cb9b-8v75t"]
Oct 30 13:51:41.112016 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:51:41.111986    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c011057-9cb5-4c50-904e-6e5c70d74637" path="/var/lib/kubelet/pods/9c011057-9cb5-4c50-904e-6e5c70d74637/volumes"
Oct 30 13:51:41.364758 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:51:41.364682    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:52:11.364770 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:52:11.364733    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:52:41.364455 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:52:41.364372    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:53:11.365433 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:53:11.365398    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:53:24.480278 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.480247    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"]
Oct 30 13:53:24.482668 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.480500    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9c011057-9cb5-4c50-904e-6e5c70d74637" containerName="console"
Oct 30 13:53:24.482668 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.480510    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c011057-9cb5-4c50-904e-6e5c70d74637" containerName="console"
Oct 30 13:53:24.482668 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.480526    4530 cpu_manager.go:414] "containerMap: removing container" podUID="051417fc-81ae-4c7e-b30d-6b0e751a6c09" containerName="insights-gathering"
Oct 30 13:53:24.482668 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.480531    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051417fc-81ae-4c7e-b30d-6b0e751a6c09" containerName="insights-gathering"
Oct 30 13:53:24.482668 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.480599    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="051417fc-81ae-4c7e-b30d-6b0e751a6c09" containerName="insights-gathering"
Oct 30 13:53:24.482668 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.480611    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c011057-9cb5-4c50-904e-6e5c70d74637" containerName="console"
Oct 30 13:53:24.483500 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.483484    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:24.515396 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.515368    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-4z5hz\""
Oct 30 13:53:24.515518 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.515371    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 13:53:24.515518 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.515454    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 13:53:24.517043 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.517019    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"]
Oct 30 13:53:24.564438 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.564403    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rmw2\" (UniqueName: \"kubernetes.io/projected/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-kube-api-access-4rmw2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47\" (UID: \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:24.564569 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.564445    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47\" (UID: \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:24.564569 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.564464    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47\" (UID: \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:24.665795 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.665757    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rmw2\" (UniqueName: \"kubernetes.io/projected/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-kube-api-access-4rmw2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47\" (UID: \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:24.665795 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.665802    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47\" (UID: \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:24.665967 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.665820    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47\" (UID: \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:24.666118 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.666102    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47\" (UID: \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:24.666206 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.666187    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47\" (UID: \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:24.706554 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.706521    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rmw2\" (UniqueName: \"kubernetes.io/projected/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-kube-api-access-4rmw2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47\" (UID: \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:24.798659 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:24.798637    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:25.094663 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:25.094636    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"]
Oct 30 13:53:25.095498 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:53:25.095479    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8ef2cd_2b6f_44b0_93bb_4e8b46b91bda.slice/crio-d033e743e84440268d3292761877288c97e7d1ccd112bee3c3657d478a45a588 WatchSource:0}: Error finding container d033e743e84440268d3292761877288c97e7d1ccd112bee3c3657d478a45a588: Status 404 returned error can't find the container with id d033e743e84440268d3292761877288c97e7d1ccd112bee3c3657d478a45a588
Oct 30 13:53:26.096787 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:26.096732    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47" event={"ID":"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda","Type":"ContainerStarted","Data":"d033e743e84440268d3292761877288c97e7d1ccd112bee3c3657d478a45a588"}
Oct 30 13:53:31.118611 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:31.118580    4530 generic.go:356] "Generic (PLEG): container finished" podID="dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" containerID="96f6abf480fb91c230bd35020d5f5f3dcd158ad7dc540a4f4ffb9f449eaf31c8" exitCode=0
Oct 30 13:53:31.118982 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:31.118637    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47" event={"ID":"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda","Type":"ContainerDied","Data":"96f6abf480fb91c230bd35020d5f5f3dcd158ad7dc540a4f4ffb9f449eaf31c8"}
Oct 30 13:53:39.146903 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:39.146871    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47" event={"ID":"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda","Type":"ContainerStarted","Data":"82539ffb6309233e4adadaafa6f5e5df01a27fe4afb7b94b3aa1632b7c8d2284"}
Oct 30 13:53:40.152067 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:40.152036    4530 generic.go:356] "Generic (PLEG): container finished" podID="dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" containerID="82539ffb6309233e4adadaafa6f5e5df01a27fe4afb7b94b3aa1632b7c8d2284" exitCode=0
Oct 30 13:53:40.152441 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:40.152079    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47" event={"ID":"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda","Type":"ContainerDied","Data":"82539ffb6309233e4adadaafa6f5e5df01a27fe4afb7b94b3aa1632b7c8d2284"}
Oct 30 13:53:41.371575 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:53:41.371513    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:53:46.174829 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:46.174753    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47" event={"ID":"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda","Type":"ContainerStarted","Data":"4496a9463fd8fe853a95fd1dfc991dc0ea77e1321f5302a79f7d52aa1fc74ce5"}
Oct 30 13:53:47.179738 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:47.179701    4530 generic.go:356] "Generic (PLEG): container finished" podID="dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" containerID="4496a9463fd8fe853a95fd1dfc991dc0ea77e1321f5302a79f7d52aa1fc74ce5" exitCode=0
Oct 30 13:53:47.180095 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:47.179764    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47" event={"ID":"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda","Type":"ContainerDied","Data":"4496a9463fd8fe853a95fd1dfc991dc0ea77e1321f5302a79f7d52aa1fc74ce5"}
Oct 30 13:53:48.341974 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:48.341944    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:48.447471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:48.447416    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-util\" (UniqueName: \"kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-util\") pod \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\" (UID: \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\") "
Oct 30 13:53:48.447471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:48.447448    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-kube-api-access-4rmw2\" (UniqueName: \"kubernetes.io/projected/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-kube-api-access-4rmw2\") pod \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\" (UID: \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\") "
Oct 30 13:53:48.447678 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:48.447481    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-bundle\") pod \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\" (UID: \"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda\") "
Oct 30 13:53:48.448130 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:48.448101    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-bundle" pod "dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" (UID: "dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:48.449610 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:48.449580    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-kube-api-access-4rmw2" pod "dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" (UID: "dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda"). InnerVolumeSpecName "kube-api-access-4rmw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:48.451515 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:48.451494    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-util" pod "dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" (UID: "dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:48.548675 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:48.548653    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rmw2\" (UniqueName: \"kubernetes.io/projected/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-kube-api-access-4rmw2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:53:48.548675 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:48.548673    4530 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:53:48.548792 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:48.548682    4530 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:53:49.187676 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:49.187652    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47"
Oct 30 13:53:49.187800 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:49.187644    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdwb47" event={"ID":"dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda","Type":"ContainerDied","Data":"d033e743e84440268d3292761877288c97e7d1ccd112bee3c3657d478a45a588"}
Oct 30 13:53:49.187800 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:49.187768    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d033e743e84440268d3292761877288c97e7d1ccd112bee3c3657d478a45a588"
Oct 30 13:53:52.603009 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.602978    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df"]
Oct 30 13:53:52.603448 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.603231    4530 cpu_manager.go:414] "containerMap: removing container" podUID="dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" containerName="util"
Oct 30 13:53:52.603448 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.603240    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" containerName="util"
Oct 30 13:53:52.603448 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.603251    4530 cpu_manager.go:414] "containerMap: removing container" podUID="dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" containerName="extract"
Oct 30 13:53:52.603448 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.603256    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" containerName="extract"
Oct 30 13:53:52.603448 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.603265    4530 cpu_manager.go:414] "containerMap: removing container" podUID="dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" containerName="pull"
Oct 30 13:53:52.603448 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.603270    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" containerName="pull"
Oct 30 13:53:52.603448 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.603315    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8ef2cd-2b6f-44b0-93bb-4e8b46b91bda" containerName="extract"
Oct 30 13:53:52.605657 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.605640    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df"
Oct 30 13:53:52.619141 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.619119    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df"]
Oct 30 13:53:52.632373 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.632346    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:53:52.632491 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.632346    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 13:53:52.632491 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.632445    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rtzfc\""
Oct 30 13:53:52.677528 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.677504    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfed052f-0733-44e1-bcd3-8350722e3e10-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-6p8df\" (UID: \"bfed052f-0733-44e1-bcd3-8350722e3e10\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df"
Oct 30 13:53:52.677657 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.677555    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b884p\" (UniqueName: \"kubernetes.io/projected/bfed052f-0733-44e1-bcd3-8350722e3e10-kube-api-access-b884p\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-6p8df\" (UID: \"bfed052f-0733-44e1-bcd3-8350722e3e10\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df"
Oct 30 13:53:52.778726 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.778704    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfed052f-0733-44e1-bcd3-8350722e3e10-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-6p8df\" (UID: \"bfed052f-0733-44e1-bcd3-8350722e3e10\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df"
Oct 30 13:53:52.778802 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.778738    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b884p\" (UniqueName: \"kubernetes.io/projected/bfed052f-0733-44e1-bcd3-8350722e3e10-kube-api-access-b884p\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-6p8df\" (UID: \"bfed052f-0733-44e1-bcd3-8350722e3e10\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df"
Oct 30 13:53:52.779043 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.779024    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfed052f-0733-44e1-bcd3-8350722e3e10-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-6p8df\" (UID: \"bfed052f-0733-44e1-bcd3-8350722e3e10\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df"
Oct 30 13:53:52.805474 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.805447    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b884p\" (UniqueName: \"kubernetes.io/projected/bfed052f-0733-44e1-bcd3-8350722e3e10-kube-api-access-b884p\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-6p8df\" (UID: \"bfed052f-0733-44e1-bcd3-8350722e3e10\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df"
Oct 30 13:53:52.915757 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:52.915708    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df"
Oct 30 13:53:53.238416 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:53.238387    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df"]
Oct 30 13:53:53.239031 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:53:53.239006    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfed052f_0733_44e1_bcd3_8350722e3e10.slice/crio-b9d2501a51eba1d0021722cd57e58b267ccc858f5bd22fe029a279f95db2fece WatchSource:0}: Error finding container b9d2501a51eba1d0021722cd57e58b267ccc858f5bd22fe029a279f95db2fece: Status 404 returned error can't find the container with id b9d2501a51eba1d0021722cd57e58b267ccc858f5bd22fe029a279f95db2fece
Oct 30 13:53:54.207438 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:54.207401    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df" event={"ID":"bfed052f-0733-44e1-bcd3-8350722e3e10","Type":"ContainerStarted","Data":"b9d2501a51eba1d0021722cd57e58b267ccc858f5bd22fe029a279f95db2fece"}
Oct 30 13:53:55.212267 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:55.212228    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df" event={"ID":"bfed052f-0733-44e1-bcd3-8350722e3e10","Type":"ContainerStarted","Data":"2ed62a4194741f9c95c0a6c00e06c4e88236437800a590a93204907b63bde5ca"}
Oct 30 13:53:55.293563 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:55.293493    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-6p8df" podStartSLOduration=1.9227260419999999 podStartE2EDuration="3.293480782s" totalImagesPullingTime="1.37075474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:55.278867275 +0000 UTC m=+300.310766213" watchObservedRunningTime="2026-10-30 13:53:55.293480782 +0000 UTC m=+300.325379723"
Oct 30 13:53:56.493005 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.492974    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"]
Oct 30 13:53:56.495353 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.495337    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:53:56.516493 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.516453    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-jvcwp\""
Oct 30 13:53:56.530786 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.530744    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 13:53:56.530786 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.530771    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 13:53:56.532086 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.532060    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"]
Oct 30 13:53:56.606523 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.606496    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj\" (UID: \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:53:56.606643 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.606564    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cpnp\" (UniqueName: \"kubernetes.io/projected/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-kube-api-access-5cpnp\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj\" (UID: \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:53:56.606711 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.606684    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj\" (UID: \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:53:56.707175 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.707151    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj\" (UID: \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:53:56.707280 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.707195    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cpnp\" (UniqueName: \"kubernetes.io/projected/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-kube-api-access-5cpnp\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj\" (UID: \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:53:56.707280 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.707217    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj\" (UID: \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:53:56.707561 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.707524    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj\" (UID: \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:53:56.707633 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.707583    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj\" (UID: \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:53:56.735092 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.735067    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cpnp\" (UniqueName: \"kubernetes.io/projected/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-kube-api-access-5cpnp\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj\" (UID: \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:53:56.807399 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:56.807378    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:53:57.103105 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:57.103081    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"]
Oct 30 13:53:57.104495 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:53:57.104459    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1ae957_6ca2_4bd4_93d0_3e302ea193ad.slice/crio-f3d45b3cbed8977f7557a7aab10a3508b92e855dabe2a0a3762afea2ef12116c WatchSource:0}: Error finding container f3d45b3cbed8977f7557a7aab10a3508b92e855dabe2a0a3762afea2ef12116c: Status 404 returned error can't find the container with id f3d45b3cbed8977f7557a7aab10a3508b92e855dabe2a0a3762afea2ef12116c
Oct 30 13:53:57.219995 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:57.219969    4530 generic.go:356] "Generic (PLEG): container finished" podID="3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" containerID="bc75e81f6c64ed2c2f218df3bb2733d1c29d9e72bf507ac72055c245b9270d9a" exitCode=0
Oct 30 13:53:57.220102 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:57.220047    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj" event={"ID":"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad","Type":"ContainerDied","Data":"bc75e81f6c64ed2c2f218df3bb2733d1c29d9e72bf507ac72055c245b9270d9a"}
Oct 30 13:53:57.220102 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:57.220094    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj" event={"ID":"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad","Type":"ContainerStarted","Data":"f3d45b3cbed8977f7557a7aab10a3508b92e855dabe2a0a3762afea2ef12116c"}
Oct 30 13:53:57.221046 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:57.221031    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:53:58.358295 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.358260    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4"]
Oct 30 13:53:58.360642 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.360607    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4"
Oct 30 13:53:58.381161 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.381136    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4"]
Oct 30 13:53:58.393770 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.393742    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 13:53:58.393957 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.393936    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bcp6x\""
Oct 30 13:53:58.393957 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.393942    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 13:53:58.520413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.520371    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlw7l\" (UniqueName: \"kubernetes.io/projected/3fc47faa-c28e-45d4-a4b9-995c238a348b-kube-api-access-wlw7l\") pod \"cert-manager-cainjector-5c57bbd4fc-9tqv4\" (UID: \"3fc47faa-c28e-45d4-a4b9-995c238a348b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4"
Oct 30 13:53:58.520413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.520416    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fc47faa-c28e-45d4-a4b9-995c238a348b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9tqv4\" (UID: \"3fc47faa-c28e-45d4-a4b9-995c238a348b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4"
Oct 30 13:53:58.621247 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.621177    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlw7l\" (UniqueName: \"kubernetes.io/projected/3fc47faa-c28e-45d4-a4b9-995c238a348b-kube-api-access-wlw7l\") pod \"cert-manager-cainjector-5c57bbd4fc-9tqv4\" (UID: \"3fc47faa-c28e-45d4-a4b9-995c238a348b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4"
Oct 30 13:53:58.621247 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.621229    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fc47faa-c28e-45d4-a4b9-995c238a348b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9tqv4\" (UID: \"3fc47faa-c28e-45d4-a4b9-995c238a348b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4"
Oct 30 13:53:58.648603 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.648573    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fc47faa-c28e-45d4-a4b9-995c238a348b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9tqv4\" (UID: \"3fc47faa-c28e-45d4-a4b9-995c238a348b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4"
Oct 30 13:53:58.648731 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.648634    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlw7l\" (UniqueName: \"kubernetes.io/projected/3fc47faa-c28e-45d4-a4b9-995c238a348b-kube-api-access-wlw7l\") pod \"cert-manager-cainjector-5c57bbd4fc-9tqv4\" (UID: \"3fc47faa-c28e-45d4-a4b9-995c238a348b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4"
Oct 30 13:53:58.684360 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.684334    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4"
Oct 30 13:53:58.996567 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:58.991396    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4"]
Oct 30 13:53:59.234221 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:59.234188    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj" event={"ID":"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad","Type":"ContainerStarted","Data":"aa13f2f66b2f2f24ee31b9f73fcf4028bea5a37e46763dcc9c7ef22b32b7a857"}
Oct 30 13:53:59.235398 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:53:59.235371    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4" event={"ID":"3fc47faa-c28e-45d4-a4b9-995c238a348b","Type":"ContainerStarted","Data":"18a274bf7e07997273ed30e26ec9b6c70cdaa22255a110d85ad7f05b5fdcab59"}
Oct 30 13:54:00.242183 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:00.242138    4530 generic.go:356] "Generic (PLEG): container finished" podID="3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" containerID="aa13f2f66b2f2f24ee31b9f73fcf4028bea5a37e46763dcc9c7ef22b32b7a857" exitCode=0
Oct 30 13:54:00.242604 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:00.242190    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj" event={"ID":"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad","Type":"ContainerDied","Data":"aa13f2f66b2f2f24ee31b9f73fcf4028bea5a37e46763dcc9c7ef22b32b7a857"}
Oct 30 13:54:01.248766 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:01.248731    4530 generic.go:356] "Generic (PLEG): container finished" podID="3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" containerID="f91a04b3ce0c797a810ac6116b27f9eaeb3d4929a289ae9742b19ff4c401f80d" exitCode=0
Oct 30 13:54:01.249203 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:01.248806    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj" event={"ID":"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad","Type":"ContainerDied","Data":"f91a04b3ce0c797a810ac6116b27f9eaeb3d4929a289ae9742b19ff4c401f80d"}
Oct 30 13:54:02.254689 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:02.254655    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4" event={"ID":"3fc47faa-c28e-45d4-a4b9-995c238a348b","Type":"ContainerStarted","Data":"9b73714fed3ea403eb71d95242be660319db198e8e5bf88344130f5eab3a876a"}
Oct 30 13:54:02.300299 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:02.300245    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9tqv4" podStartSLOduration=1.895631676 podStartE2EDuration="4.300231751s" totalImagesPullingTime="2.404600075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:02.299843036 +0000 UTC m=+307.331741976" watchObservedRunningTime="2026-10-30 13:54:02.300231751 +0000 UTC m=+307.332130689"
Oct 30 13:54:02.375607 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:02.375586    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:54:02.552767 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:02.552746    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-util\" (UniqueName: \"kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-util\") pod \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\" (UID: \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\") "
Oct 30 13:54:02.552927 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:02.552801    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-bundle\") pod \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\" (UID: \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\") "
Oct 30 13:54:02.552927 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:02.552832    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-kube-api-access-5cpnp\" (UniqueName: \"kubernetes.io/projected/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-kube-api-access-5cpnp\") pod \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\" (UID: \"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad\") "
Oct 30 13:54:02.553227 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:02.553205    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-bundle" pod "3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" (UID: "3d1ae957-6ca2-4bd4-93d0-3e302ea193ad"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:02.554963 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:02.554943    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-kube-api-access-5cpnp" pod "3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" (UID: "3d1ae957-6ca2-4bd4-93d0-3e302ea193ad"). InnerVolumeSpecName "kube-api-access-5cpnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:02.557827 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:02.557803    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-util" pod "3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" (UID: "3d1ae957-6ca2-4bd4-93d0-3e302ea193ad"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:02.653906 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:02.653882    4530 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:54:02.653906 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:02.653903    4530 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:54:02.654028 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:02.653913    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cpnp\" (UniqueName: \"kubernetes.io/projected/3d1ae957-6ca2-4bd4-93d0-3e302ea193ad-kube-api-access-5cpnp\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:54:03.260122 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:03.260097    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj"
Oct 30 13:54:03.260466 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:03.260112    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85thdfj" event={"ID":"3d1ae957-6ca2-4bd4-93d0-3e302ea193ad","Type":"ContainerDied","Data":"f3d45b3cbed8977f7557a7aab10a3508b92e855dabe2a0a3762afea2ef12116c"}
Oct 30 13:54:03.260466 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:03.260158    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3d45b3cbed8977f7557a7aab10a3508b92e855dabe2a0a3762afea2ef12116c"
Oct 30 13:54:09.421531 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.421453    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq"]
Oct 30 13:54:09.421889 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.421728    4530 cpu_manager.go:414] "containerMap: removing container" podUID="3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" containerName="util"
Oct 30 13:54:09.421889 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.421739    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" containerName="util"
Oct 30 13:54:09.421889 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.421747    4530 cpu_manager.go:414] "containerMap: removing container" podUID="3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" containerName="extract"
Oct 30 13:54:09.421889 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.421752    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" containerName="extract"
Oct 30 13:54:09.421889 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.421768    4530 cpu_manager.go:414] "containerMap: removing container" podUID="3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" containerName="pull"
Oct 30 13:54:09.421889 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.421773    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" containerName="pull"
Oct 30 13:54:09.421889 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.421825    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1ae957-6ca2-4bd4-93d0-3e302ea193ad" containerName="extract"
Oct 30 13:54:09.424460 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.424443    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq"
Oct 30 13:54:09.456419 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.456397    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:54:09.456529 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.456458    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-tt2qw\""
Oct 30 13:54:09.456529 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.456467    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 13:54:09.457428 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.457411    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq"]
Oct 30 13:54:09.604607 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.604582    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d139aecb-0f40-4c04-a344-29d2a6a6ef9e-tmp\") pod \"openshift-lws-operator-8465994787-6r2jq\" (UID: \"d139aecb-0f40-4c04-a344-29d2a6a6ef9e\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq"
Oct 30 13:54:09.604715 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.604624    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mspf4\" (UniqueName: \"kubernetes.io/projected/d139aecb-0f40-4c04-a344-29d2a6a6ef9e-kube-api-access-mspf4\") pod \"openshift-lws-operator-8465994787-6r2jq\" (UID: \"d139aecb-0f40-4c04-a344-29d2a6a6ef9e\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq"
Oct 30 13:54:09.705363 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.705306    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mspf4\" (UniqueName: \"kubernetes.io/projected/d139aecb-0f40-4c04-a344-29d2a6a6ef9e-kube-api-access-mspf4\") pod \"openshift-lws-operator-8465994787-6r2jq\" (UID: \"d139aecb-0f40-4c04-a344-29d2a6a6ef9e\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq"
Oct 30 13:54:09.705466 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.705382    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d139aecb-0f40-4c04-a344-29d2a6a6ef9e-tmp\") pod \"openshift-lws-operator-8465994787-6r2jq\" (UID: \"d139aecb-0f40-4c04-a344-29d2a6a6ef9e\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq"
Oct 30 13:54:09.705727 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.705708    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d139aecb-0f40-4c04-a344-29d2a6a6ef9e-tmp\") pod \"openshift-lws-operator-8465994787-6r2jq\" (UID: \"d139aecb-0f40-4c04-a344-29d2a6a6ef9e\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq"
Oct 30 13:54:09.735780 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:09.735760    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mspf4\" (UniqueName: \"kubernetes.io/projected/d139aecb-0f40-4c04-a344-29d2a6a6ef9e-kube-api-access-mspf4\") pod \"openshift-lws-operator-8465994787-6r2jq\" (UID: \"d139aecb-0f40-4c04-a344-29d2a6a6ef9e\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq"
Oct 30 13:54:10.035092 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:10.035035    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq"
Oct 30 13:54:10.329688 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:10.329663    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq"]
Oct 30 13:54:10.330175 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:54:10.330152    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd139aecb_0f40_4c04_a344_29d2a6a6ef9e.slice/crio-2bd480128815a51acc730060d2946d4e8ffa68c9ebff997b2148b95ec578a4aa WatchSource:0}: Error finding container 2bd480128815a51acc730060d2946d4e8ffa68c9ebff997b2148b95ec578a4aa: Status 404 returned error can't find the container with id 2bd480128815a51acc730060d2946d4e8ffa68c9ebff997b2148b95ec578a4aa
Oct 30 13:54:11.293704 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:11.293672    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq" event={"ID":"d139aecb-0f40-4c04-a344-29d2a6a6ef9e","Type":"ContainerStarted","Data":"2bd480128815a51acc730060d2946d4e8ffa68c9ebff997b2148b95ec578a4aa"}
Oct 30 13:54:11.365688 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:54:11.365654    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:54:21.334819 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:21.334781    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq" event={"ID":"d139aecb-0f40-4c04-a344-29d2a6a6ef9e","Type":"ContainerStarted","Data":"fdc7fce7e0b064ce1f2584780e05655f266f311b06f43a60df4b1c509b986af7"}
Oct 30 13:54:21.381739 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:21.381694    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-6r2jq" podStartSLOduration=2.143471063 podStartE2EDuration="12.38168192s" totalImagesPullingTime="10.238210857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:21.380652882 +0000 UTC m=+326.412551822" watchObservedRunningTime="2026-10-30 13:54:21.38168192 +0000 UTC m=+326.413580859"
Oct 30 13:54:25.278410 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.278376    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"]
Oct 30 13:54:25.281022 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.281000    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:25.306055 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.306023    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"]
Oct 30 13:54:25.312561 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.312529    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg\" (UID: \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:25.312669 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.312612    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg\" (UID: \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:25.312719 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.312661    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7s2v\" (UniqueName: \"kubernetes.io/projected/b9d56842-22b5-45ed-b1e9-5d1bc4686454-kube-api-access-z7s2v\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg\" (UID: \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:25.319459 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.319439    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 13:54:25.319567 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.319469    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 13:54:25.319567 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.319506    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-d4qhp\""
Oct 30 13:54:25.413331 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.413303    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg\" (UID: \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:25.413424 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.413367    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg\" (UID: \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:25.413424 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.413387    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7s2v\" (UniqueName: \"kubernetes.io/projected/b9d56842-22b5-45ed-b1e9-5d1bc4686454-kube-api-access-z7s2v\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg\" (UID: \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:25.413688 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.413671    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg\" (UID: \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:25.413721 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.413709    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg\" (UID: \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:25.440337 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.440311    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7s2v\" (UniqueName: \"kubernetes.io/projected/b9d56842-22b5-45ed-b1e9-5d1bc4686454-kube-api-access-z7s2v\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg\" (UID: \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:25.593120 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.593098    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:25.889402 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:25.889378    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"]
Oct 30 13:54:25.890625 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:54:25.890601    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d56842_22b5_45ed_b1e9_5d1bc4686454.slice/crio-bca173120356bf70950c4a7b23caea0b905944ed6b356e098a73d832c50fe526 WatchSource:0}: Error finding container bca173120356bf70950c4a7b23caea0b905944ed6b356e098a73d832c50fe526: Status 404 returned error can't find the container with id bca173120356bf70950c4a7b23caea0b905944ed6b356e098a73d832c50fe526
Oct 30 13:54:26.356119 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:26.356091    4530 generic.go:356] "Generic (PLEG): container finished" podID="b9d56842-22b5-45ed-b1e9-5d1bc4686454" containerID="df74bdbda4647e02b1e7aeb90a67f5456fa6633e69a957ea2362c7c5d4e35374" exitCode=0
Oct 30 13:54:26.356488 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:26.356148    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg" event={"ID":"b9d56842-22b5-45ed-b1e9-5d1bc4686454","Type":"ContainerDied","Data":"df74bdbda4647e02b1e7aeb90a67f5456fa6633e69a957ea2362c7c5d4e35374"}
Oct 30 13:54:26.356488 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:26.356181    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg" event={"ID":"b9d56842-22b5-45ed-b1e9-5d1bc4686454","Type":"ContainerStarted","Data":"bca173120356bf70950c4a7b23caea0b905944ed6b356e098a73d832c50fe526"}
Oct 30 13:54:27.361805 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:27.361774    4530 generic.go:356] "Generic (PLEG): container finished" podID="b9d56842-22b5-45ed-b1e9-5d1bc4686454" containerID="38d6ad5b7063b1fdc5ebb249e9b109044a1f4fa632a0dd5d2b8f71237cf19571" exitCode=0
Oct 30 13:54:27.362137 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:27.361857    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg" event={"ID":"b9d56842-22b5-45ed-b1e9-5d1bc4686454","Type":"ContainerDied","Data":"38d6ad5b7063b1fdc5ebb249e9b109044a1f4fa632a0dd5d2b8f71237cf19571"}
Oct 30 13:54:28.368905 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:28.368870    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg" event={"ID":"b9d56842-22b5-45ed-b1e9-5d1bc4686454","Type":"ContainerStarted","Data":"958d5fad1a59b764f6877abc16184954c12cbb352b7448a5e033179b949772a2"}
Oct 30 13:54:28.417714 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:28.417662    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg" podStartSLOduration=2.821624625 podStartE2EDuration="3.417647742s" totalImagesPullingTime="596.023117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:28.417579771 +0000 UTC m=+333.449478710" watchObservedRunningTime="2026-10-30 13:54:28.417647742 +0000 UTC m=+333.449546681"
Oct 30 13:54:38.406908 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:38.406880    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 13:54:38.407569 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:38.407528    4530 generic.go:356] "Generic (PLEG): container finished" podID="b9d56842-22b5-45ed-b1e9-5d1bc4686454" containerID="958d5fad1a59b764f6877abc16184954c12cbb352b7448a5e033179b949772a2" exitCode=1
Oct 30 13:54:38.407670 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:38.407570    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg" event={"ID":"b9d56842-22b5-45ed-b1e9-5d1bc4686454","Type":"ContainerDied","Data":"958d5fad1a59b764f6877abc16184954c12cbb352b7448a5e033179b949772a2"}
Oct 30 13:54:39.528192 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:39.528169    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 13:54:39.530730 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:39.530711    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:39.610742 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:39.610717    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-bundle\") pod \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\" (UID: \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\") "
Oct 30 13:54:39.610840 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:39.610753    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-util\" (UniqueName: \"kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-util\") pod \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\" (UID: \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\") "
Oct 30 13:54:39.610840 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:39.610778    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d56842-22b5-45ed-b1e9-5d1bc4686454-kube-api-access-z7s2v\" (UniqueName: \"kubernetes.io/projected/b9d56842-22b5-45ed-b1e9-5d1bc4686454-kube-api-access-z7s2v\") pod \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\" (UID: \"b9d56842-22b5-45ed-b1e9-5d1bc4686454\") "
Oct 30 13:54:39.611281 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:39.611259    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-bundle" pod "b9d56842-22b5-45ed-b1e9-5d1bc4686454" (UID: "b9d56842-22b5-45ed-b1e9-5d1bc4686454"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:39.612860 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:39.612832    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d56842-22b5-45ed-b1e9-5d1bc4686454-kube-api-access-z7s2v" pod "b9d56842-22b5-45ed-b1e9-5d1bc4686454" (UID: "b9d56842-22b5-45ed-b1e9-5d1bc4686454"). InnerVolumeSpecName "kube-api-access-z7s2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:39.615242 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:39.615223    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-util" pod "b9d56842-22b5-45ed-b1e9-5d1bc4686454" (UID: "b9d56842-22b5-45ed-b1e9-5d1bc4686454"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:39.711580 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:39.711514    4530 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:54:39.711580 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:39.711536    4530 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9d56842-22b5-45ed-b1e9-5d1bc4686454-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:54:39.711580 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:39.711559    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7s2v\" (UniqueName: \"kubernetes.io/projected/b9d56842-22b5-45ed-b1e9-5d1bc4686454-kube-api-access-z7s2v\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:54:40.416773 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:40.416746    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 13:54:40.417509 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:40.417479    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg" event={"ID":"b9d56842-22b5-45ed-b1e9-5d1bc4686454","Type":"ContainerDied","Data":"bca173120356bf70950c4a7b23caea0b905944ed6b356e098a73d832c50fe526"}
Oct 30 13:54:40.417644 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:40.417525    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bca173120356bf70950c4a7b23caea0b905944ed6b356e098a73d832c50fe526"
Oct 30 13:54:40.417644 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:40.417530    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg"
Oct 30 13:54:41.365136 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:54:41.365104    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:54:47.036619 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.036589    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"]
Oct 30 13:54:47.036981 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.036877    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b9d56842-22b5-45ed-b1e9-5d1bc4686454" containerName="extract"
Oct 30 13:54:47.036981 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.036888    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d56842-22b5-45ed-b1e9-5d1bc4686454" containerName="extract"
Oct 30 13:54:47.036981 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.036902    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b9d56842-22b5-45ed-b1e9-5d1bc4686454" containerName="util"
Oct 30 13:54:47.036981 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.036907    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d56842-22b5-45ed-b1e9-5d1bc4686454" containerName="util"
Oct 30 13:54:47.036981 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.036920    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b9d56842-22b5-45ed-b1e9-5d1bc4686454" containerName="pull"
Oct 30 13:54:47.036981 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.036925    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d56842-22b5-45ed-b1e9-5d1bc4686454" containerName="pull"
Oct 30 13:54:47.037186 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.036994    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d56842-22b5-45ed-b1e9-5d1bc4686454" containerName="extract"
Oct 30 13:54:47.040115 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.040091    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:54:47.058027 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.057998    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl\" (UID: \"bab1137f-277b-48ec-81b0-60533ca6d832\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:54:47.058141 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.058048    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm4pz\" (UniqueName: \"kubernetes.io/projected/bab1137f-277b-48ec-81b0-60533ca6d832-kube-api-access-qm4pz\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl\" (UID: \"bab1137f-277b-48ec-81b0-60533ca6d832\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:54:47.058196 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.058180    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl\" (UID: \"bab1137f-277b-48ec-81b0-60533ca6d832\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:54:47.072328 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.072305    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 13:54:47.072401 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.072332    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 13:54:47.072475 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.072460    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-d4qhp\""
Oct 30 13:54:47.074259 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.074240    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"]
Oct 30 13:54:47.159280 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.159254    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl\" (UID: \"bab1137f-277b-48ec-81b0-60533ca6d832\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:54:47.159371 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.159290    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl\" (UID: \"bab1137f-277b-48ec-81b0-60533ca6d832\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:54:47.159371 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.159321    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm4pz\" (UniqueName: \"kubernetes.io/projected/bab1137f-277b-48ec-81b0-60533ca6d832-kube-api-access-qm4pz\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl\" (UID: \"bab1137f-277b-48ec-81b0-60533ca6d832\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:54:47.159641 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.159625    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl\" (UID: \"bab1137f-277b-48ec-81b0-60533ca6d832\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:54:47.159711 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.159696    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl\" (UID: \"bab1137f-277b-48ec-81b0-60533ca6d832\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:54:47.194255 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.194226    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm4pz\" (UniqueName: \"kubernetes.io/projected/bab1137f-277b-48ec-81b0-60533ca6d832-kube-api-access-qm4pz\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl\" (UID: \"bab1137f-277b-48ec-81b0-60533ca6d832\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:54:47.356211 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.356181    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:54:47.650582 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:47.650556    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"]
Oct 30 13:54:47.651509 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:54:47.651488    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab1137f_277b_48ec_81b0_60533ca6d832.slice/crio-22ce1243ed457877110e7f020330e95837639b84ed01159541c0d16b5133440c WatchSource:0}: Error finding container 22ce1243ed457877110e7f020330e95837639b84ed01159541c0d16b5133440c: Status 404 returned error can't find the container with id 22ce1243ed457877110e7f020330e95837639b84ed01159541c0d16b5133440c
Oct 30 13:54:48.449644 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:48.449578    4530 generic.go:356] "Generic (PLEG): container finished" podID="bab1137f-277b-48ec-81b0-60533ca6d832" containerID="423df3bc47c0d41e00d39d222a1558ac376427d8feae4cd7aa243a4ddafd9b2a" exitCode=0
Oct 30 13:54:48.449644 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:48.449617    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl" event={"ID":"bab1137f-277b-48ec-81b0-60533ca6d832","Type":"ContainerDied","Data":"423df3bc47c0d41e00d39d222a1558ac376427d8feae4cd7aa243a4ddafd9b2a"}
Oct 30 13:54:48.450022 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:48.449651    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl" event={"ID":"bab1137f-277b-48ec-81b0-60533ca6d832","Type":"ContainerStarted","Data":"22ce1243ed457877110e7f020330e95837639b84ed01159541c0d16b5133440c"}
Oct 30 13:54:49.455670 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:49.455640    4530 generic.go:356] "Generic (PLEG): container finished" podID="bab1137f-277b-48ec-81b0-60533ca6d832" containerID="32c693bda2d6bbdad1c3028461b713e580eda8af3a4f24ff16d1ff41635e916c" exitCode=0
Oct 30 13:54:49.456050 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:49.455693    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl" event={"ID":"bab1137f-277b-48ec-81b0-60533ca6d832","Type":"ContainerDied","Data":"32c693bda2d6bbdad1c3028461b713e580eda8af3a4f24ff16d1ff41635e916c"}
Oct 30 13:54:50.461553 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:50.461504    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl" event={"ID":"bab1137f-277b-48ec-81b0-60533ca6d832","Type":"ContainerStarted","Data":"824030de3ba5836f899c29afcf7e5490a4da95d1f507ce665b463b5cf4e146c7"}
Oct 30 13:54:50.514098 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:54:50.514050    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl" podStartSLOduration=3.214562423 podStartE2EDuration="3.514037181s" totalImagesPullingTime="299.474758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:50.512759212 +0000 UTC m=+355.544658150" watchObservedRunningTime="2026-10-30 13:54:50.514037181 +0000 UTC m=+355.545936119"
Oct 30 13:55:00.504217 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:00.504189    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 13:55:00.504891 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:00.504866    4530 generic.go:356] "Generic (PLEG): container finished" podID="bab1137f-277b-48ec-81b0-60533ca6d832" containerID="824030de3ba5836f899c29afcf7e5490a4da95d1f507ce665b463b5cf4e146c7" exitCode=1
Oct 30 13:55:00.504997 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:00.504940    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl" event={"ID":"bab1137f-277b-48ec-81b0-60533ca6d832","Type":"ContainerDied","Data":"824030de3ba5836f899c29afcf7e5490a4da95d1f507ce665b463b5cf4e146c7"}
Oct 30 13:55:01.631313 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:01.631290    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 13:55:01.634700 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:01.634678    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:55:01.659083 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:01.659060    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-util\" (UniqueName: \"kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-util\") pod \"bab1137f-277b-48ec-81b0-60533ca6d832\" (UID: \"bab1137f-277b-48ec-81b0-60533ca6d832\") "
Oct 30 13:55:01.659196 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:01.659112    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-bundle\") pod \"bab1137f-277b-48ec-81b0-60533ca6d832\" (UID: \"bab1137f-277b-48ec-81b0-60533ca6d832\") "
Oct 30 13:55:01.659196 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:01.659160    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab1137f-277b-48ec-81b0-60533ca6d832-kube-api-access-qm4pz\" (UniqueName: \"kubernetes.io/projected/bab1137f-277b-48ec-81b0-60533ca6d832-kube-api-access-qm4pz\") pod \"bab1137f-277b-48ec-81b0-60533ca6d832\" (UID: \"bab1137f-277b-48ec-81b0-60533ca6d832\") "
Oct 30 13:55:01.659653 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:01.659627    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-bundle" pod "bab1137f-277b-48ec-81b0-60533ca6d832" (UID: "bab1137f-277b-48ec-81b0-60533ca6d832"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:01.661146 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:01.661126    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab1137f-277b-48ec-81b0-60533ca6d832-kube-api-access-qm4pz" pod "bab1137f-277b-48ec-81b0-60533ca6d832" (UID: "bab1137f-277b-48ec-81b0-60533ca6d832"). InnerVolumeSpecName "kube-api-access-qm4pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:01.663555 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:01.663518    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-util" pod "bab1137f-277b-48ec-81b0-60533ca6d832" (UID: "bab1137f-277b-48ec-81b0-60533ca6d832"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:01.759722 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:01.759666    4530 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:55:01.759722 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:01.759689    4530 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab1137f-277b-48ec-81b0-60533ca6d832-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:55:01.759722 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:01.759698    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm4pz\" (UniqueName: \"kubernetes.io/projected/bab1137f-277b-48ec-81b0-60533ca6d832-kube-api-access-qm4pz\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:55:02.514598 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:02.514568    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 13:55:02.515286 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:02.515267    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl"
Oct 30 13:55:02.515286 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:02.515276    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl" event={"ID":"bab1137f-277b-48ec-81b0-60533ca6d832","Type":"ContainerDied","Data":"22ce1243ed457877110e7f020330e95837639b84ed01159541c0d16b5133440c"}
Oct 30 13:55:02.515382 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:02.515316    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22ce1243ed457877110e7f020330e95837639b84ed01159541c0d16b5133440c"
Oct 30 13:55:11.365465 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:55:11.365424    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:55:19.035425 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.035392    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"]
Oct 30 13:55:19.035822 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.035719    4530 cpu_manager.go:414] "containerMap: removing container" podUID="bab1137f-277b-48ec-81b0-60533ca6d832" containerName="extract"
Oct 30 13:55:19.035822 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.035731    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab1137f-277b-48ec-81b0-60533ca6d832" containerName="extract"
Oct 30 13:55:19.035822 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.035744    4530 cpu_manager.go:414] "containerMap: removing container" podUID="bab1137f-277b-48ec-81b0-60533ca6d832" containerName="pull"
Oct 30 13:55:19.035822 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.035751    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab1137f-277b-48ec-81b0-60533ca6d832" containerName="pull"
Oct 30 13:55:19.035822 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.035792    4530 cpu_manager.go:414] "containerMap: removing container" podUID="bab1137f-277b-48ec-81b0-60533ca6d832" containerName="util"
Oct 30 13:55:19.035822 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.035797    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab1137f-277b-48ec-81b0-60533ca6d832" containerName="util"
Oct 30 13:55:19.036002 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.035848    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab1137f-277b-48ec-81b0-60533ca6d832" containerName="extract"
Oct 30 13:55:19.038808 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.038792    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:19.057441 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.057414    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"]
Oct 30 13:55:19.070286 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.070264    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 13:55:19.070404 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.070344    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 13:55:19.070404 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.070385    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-d4qhp\""
Oct 30 13:55:19.182834 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.182805    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4fnf\" (UniqueName: \"kubernetes.io/projected/29395986-2a17-4a49-9658-82a2d37543af-kube-api-access-l4fnf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl\" (UID: \"29395986-2a17-4a49-9658-82a2d37543af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:19.182945 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.182914    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl\" (UID: \"29395986-2a17-4a49-9658-82a2d37543af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:19.182999 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.182956    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl\" (UID: \"29395986-2a17-4a49-9658-82a2d37543af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:19.284184 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.284161    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl\" (UID: \"29395986-2a17-4a49-9658-82a2d37543af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:19.284286 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.284196    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl\" (UID: \"29395986-2a17-4a49-9658-82a2d37543af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:19.284286 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.284221    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4fnf\" (UniqueName: \"kubernetes.io/projected/29395986-2a17-4a49-9658-82a2d37543af-kube-api-access-l4fnf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl\" (UID: \"29395986-2a17-4a49-9658-82a2d37543af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:19.284534 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.284515    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl\" (UID: \"29395986-2a17-4a49-9658-82a2d37543af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:19.284589 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.284563    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl\" (UID: \"29395986-2a17-4a49-9658-82a2d37543af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:19.311066 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.311040    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4fnf\" (UniqueName: \"kubernetes.io/projected/29395986-2a17-4a49-9658-82a2d37543af-kube-api-access-l4fnf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl\" (UID: \"29395986-2a17-4a49-9658-82a2d37543af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:19.354125 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.354103    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:19.649497 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:19.649465    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"]
Oct 30 13:55:19.649620 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:55:19.649476    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29395986_2a17_4a49_9658_82a2d37543af.slice/crio-aff39c680653ef0b98560cd0f0accc4418a58e24c3e37d89ba0137407abde8cf WatchSource:0}: Error finding container aff39c680653ef0b98560cd0f0accc4418a58e24c3e37d89ba0137407abde8cf: Status 404 returned error can't find the container with id aff39c680653ef0b98560cd0f0accc4418a58e24c3e37d89ba0137407abde8cf
Oct 30 13:55:20.585682 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:20.585600    4530 generic.go:356] "Generic (PLEG): container finished" podID="29395986-2a17-4a49-9658-82a2d37543af" containerID="f48ea56fb2a36e848ea4399e45a6c87184bde3b3d0f5af088dd192ab6a0324f9" exitCode=0
Oct 30 13:55:20.585682 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:20.585666    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl" event={"ID":"29395986-2a17-4a49-9658-82a2d37543af","Type":"ContainerDied","Data":"f48ea56fb2a36e848ea4399e45a6c87184bde3b3d0f5af088dd192ab6a0324f9"}
Oct 30 13:55:20.586080 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:20.585710    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl" event={"ID":"29395986-2a17-4a49-9658-82a2d37543af","Type":"ContainerStarted","Data":"aff39c680653ef0b98560cd0f0accc4418a58e24c3e37d89ba0137407abde8cf"}
Oct 30 13:55:21.591321 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:21.591293    4530 generic.go:356] "Generic (PLEG): container finished" podID="29395986-2a17-4a49-9658-82a2d37543af" containerID="b1b45b88bb36640241492e873b6f8594b7c4bdc6328ff1b2a16ff771ec757d25" exitCode=0
Oct 30 13:55:21.591708 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:21.591382    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl" event={"ID":"29395986-2a17-4a49-9658-82a2d37543af","Type":"ContainerDied","Data":"b1b45b88bb36640241492e873b6f8594b7c4bdc6328ff1b2a16ff771ec757d25"}
Oct 30 13:55:22.597993 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:22.597960    4530 generic.go:356] "Generic (PLEG): container finished" podID="29395986-2a17-4a49-9658-82a2d37543af" containerID="1c85587b6e6cd33ec9abe53dc7b6aef8d188ea9cacc7be3038efee45354649d6" exitCode=0
Oct 30 13:55:22.598359 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:22.598035    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl" event={"ID":"29395986-2a17-4a49-9658-82a2d37543af","Type":"ContainerDied","Data":"1c85587b6e6cd33ec9abe53dc7b6aef8d188ea9cacc7be3038efee45354649d6"}
Oct 30 13:55:23.723990 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:23.723964    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:23.819481 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:23.819454    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-util\" (UniqueName: \"kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-util\") pod \"29395986-2a17-4a49-9658-82a2d37543af\" (UID: \"29395986-2a17-4a49-9658-82a2d37543af\") "
Oct 30 13:55:23.819624 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:23.819507    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29395986-2a17-4a49-9658-82a2d37543af-kube-api-access-l4fnf\" (UniqueName: \"kubernetes.io/projected/29395986-2a17-4a49-9658-82a2d37543af-kube-api-access-l4fnf\") pod \"29395986-2a17-4a49-9658-82a2d37543af\" (UID: \"29395986-2a17-4a49-9658-82a2d37543af\") "
Oct 30 13:55:23.819624 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:23.819527    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-bundle\" (UniqueName: \"kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-bundle\") pod \"29395986-2a17-4a49-9658-82a2d37543af\" (UID: \"29395986-2a17-4a49-9658-82a2d37543af\") "
Oct 30 13:55:23.820257 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:23.820224    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-bundle" pod "29395986-2a17-4a49-9658-82a2d37543af" (UID: "29395986-2a17-4a49-9658-82a2d37543af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:23.821582 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:23.821524    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29395986-2a17-4a49-9658-82a2d37543af-kube-api-access-l4fnf" pod "29395986-2a17-4a49-9658-82a2d37543af" (UID: "29395986-2a17-4a49-9658-82a2d37543af"). InnerVolumeSpecName "kube-api-access-l4fnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:23.825181 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:23.825082    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-util" pod "29395986-2a17-4a49-9658-82a2d37543af" (UID: "29395986-2a17-4a49-9658-82a2d37543af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:23.920579 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:23.920526    4530 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:55:23.920579 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:23.920566    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4fnf\" (UniqueName: \"kubernetes.io/projected/29395986-2a17-4a49-9658-82a2d37543af-kube-api-access-l4fnf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:55:23.920579 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:23.920576    4530 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29395986-2a17-4a49-9658-82a2d37543af-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:55:24.608384 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:24.608354    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl"
Oct 30 13:55:24.608384 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:24.608375    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z5zkl" event={"ID":"29395986-2a17-4a49-9658-82a2d37543af","Type":"ContainerDied","Data":"aff39c680653ef0b98560cd0f0accc4418a58e24c3e37d89ba0137407abde8cf"}
Oct 30 13:55:24.608601 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:24.608422    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aff39c680653ef0b98560cd0f0accc4418a58e24c3e37d89ba0137407abde8cf"
Oct 30 13:55:30.025685 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.025650    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"]
Oct 30 13:55:30.026185 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.025921    4530 cpu_manager.go:414] "containerMap: removing container" podUID="29395986-2a17-4a49-9658-82a2d37543af" containerName="pull"
Oct 30 13:55:30.026185 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.025932    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29395986-2a17-4a49-9658-82a2d37543af" containerName="pull"
Oct 30 13:55:30.026185 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.025942    4530 cpu_manager.go:414] "containerMap: removing container" podUID="29395986-2a17-4a49-9658-82a2d37543af" containerName="extract"
Oct 30 13:55:30.026185 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.025946    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29395986-2a17-4a49-9658-82a2d37543af" containerName="extract"
Oct 30 13:55:30.026185 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.025954    4530 cpu_manager.go:414] "containerMap: removing container" podUID="29395986-2a17-4a49-9658-82a2d37543af" containerName="util"
Oct 30 13:55:30.026185 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.025959    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29395986-2a17-4a49-9658-82a2d37543af" containerName="util"
Oct 30 13:55:30.026185 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.026014    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="29395986-2a17-4a49-9658-82a2d37543af" containerName="extract"
Oct 30 13:55:30.029049 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.029033    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.050321 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.050293    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 13:55:30.050473 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.050331    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 13:55:30.050473 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.050372    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 13:55:30.064144 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.064121    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 13:55:30.064265 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.064197    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hnlvr\""
Oct 30 13:55:30.064495 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.064480    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 13:55:30.066328 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.066307    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"]
Oct 30 13:55:30.163679 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.163651    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2588566f-38bd-4f85-9190-2b8cce575c23-tmp\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.163821 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.163681    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2588566f-38bd-4f85-9190-2b8cce575c23-metrics-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.163821 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.163698    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2588566f-38bd-4f85-9190-2b8cce575c23-manifests\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.163821 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.163724    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2588566f-38bd-4f85-9190-2b8cce575c23-webhook-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.163979 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.163853    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2588566f-38bd-4f85-9190-2b8cce575c23-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.163979 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.163892    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7lp\" (UniqueName: \"kubernetes.io/projected/2588566f-38bd-4f85-9190-2b8cce575c23-kube-api-access-jr7lp\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.265100 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.265071    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2588566f-38bd-4f85-9190-2b8cce575c23-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.265249 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.265104    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7lp\" (UniqueName: \"kubernetes.io/projected/2588566f-38bd-4f85-9190-2b8cce575c23-kube-api-access-jr7lp\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.265249 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.265152    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2588566f-38bd-4f85-9190-2b8cce575c23-tmp\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.265249 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.265173    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2588566f-38bd-4f85-9190-2b8cce575c23-metrics-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.265249 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.265187    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2588566f-38bd-4f85-9190-2b8cce575c23-manifests\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.265249 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.265211    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2588566f-38bd-4f85-9190-2b8cce575c23-webhook-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.265696 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.265663    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2588566f-38bd-4f85-9190-2b8cce575c23-manifests\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.267628 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.267597    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2588566f-38bd-4f85-9190-2b8cce575c23-tmp\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.267778 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.267761    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2588566f-38bd-4f85-9190-2b8cce575c23-webhook-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.268125 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.268102    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2588566f-38bd-4f85-9190-2b8cce575c23-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.268187 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.268102    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2588566f-38bd-4f85-9190-2b8cce575c23-metrics-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.291818 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.291761    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7lp\" (UniqueName: \"kubernetes.io/projected/2588566f-38bd-4f85-9190-2b8cce575c23-kube-api-access-jr7lp\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-2nm4s\" (UID: \"2588566f-38bd-4f85-9190-2b8cce575c23\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.348731 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.348709    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:30.716481 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:30.716452    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"]
Oct 30 13:55:30.728688 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:55:30.728659    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2588566f_38bd_4f85_9190_2b8cce575c23.slice/crio-4fe54b3bf0b9480c495814e6837209746b95e1a4161c48dfd43c87f889c11abc WatchSource:0}: Error finding container 4fe54b3bf0b9480c495814e6837209746b95e1a4161c48dfd43c87f889c11abc: Status 404 returned error can't find the container with id 4fe54b3bf0b9480c495814e6837209746b95e1a4161c48dfd43c87f889c11abc
Oct 30 13:55:31.639311 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:31.639234    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s" event={"ID":"2588566f-38bd-4f85-9190-2b8cce575c23","Type":"ContainerStarted","Data":"4fe54b3bf0b9480c495814e6837209746b95e1a4161c48dfd43c87f889c11abc"}
Oct 30 13:55:33.650484 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:33.650457    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s" event={"ID":"2588566f-38bd-4f85-9190-2b8cce575c23","Type":"ContainerStarted","Data":"aa2bf56841399a0a036b8bb339b29281be2e5f3fd92dd5bdf702d4149a993343"}
Oct 30 13:55:34.656895 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:34.656858    4530 generic.go:356] "Generic (PLEG): container finished" podID="2588566f-38bd-4f85-9190-2b8cce575c23" containerID="aa2bf56841399a0a036b8bb339b29281be2e5f3fd92dd5bdf702d4149a993343" exitCode=0
Oct 30 13:55:34.657288 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:34.656926    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s" event={"ID":"2588566f-38bd-4f85-9190-2b8cce575c23","Type":"ContainerDied","Data":"aa2bf56841399a0a036b8bb339b29281be2e5f3fd92dd5bdf702d4149a993343"}
Oct 30 13:55:35.663158 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:35.663123    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s" event={"ID":"2588566f-38bd-4f85-9190-2b8cce575c23","Type":"ContainerStarted","Data":"acefa9bb43093f76d95a0eaf4d3c892acb0dbee4dcbbcfc52912db0a0bc8085a"}
Oct 30 13:55:35.663527 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:35.663256    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:35.759956 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:35.759906    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s" podStartSLOduration=3.131684386 podStartE2EDuration="6.759891493s" totalImagesPullingTime="3.142392608s" totalInitContainerRuntime="485.814499ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:35.744454073 +0000 UTC m=+400.776353025" watchObservedRunningTime="2026-10-30 13:55:35.759891493 +0000 UTC m=+400.791790431"
Oct 30 13:55:41.365674 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:55:41.365599    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:55:46.670639 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:46.670613    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-2nm4s"
Oct 30 13:55:52.998975 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:52.998937    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8996685d-q52jz"]
Oct 30 13:55:53.010448 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.010421    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz"
Oct 30 13:55:53.022839 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.022816    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8996685d-q52jz"]
Oct 30 13:55:53.035390 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.035359    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 13:55:53.035390 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.035375    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 13:55:53.035599 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.035366    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 13:55:53.035599 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.035376    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r6bms\""
Oct 30 13:55:53.035599 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.035363    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 13:55:53.058534 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.058507    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-k44b2"]
Oct 30 13:55:53.062648 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.062628    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-k44b2"
Oct 30 13:55:53.083979 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.083957    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-k44b2"]
Oct 30 13:55:53.121984 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.121963    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cba183dd-ad5b-469f-9058-67196f9557e5-tmp\") pod \"kube-auth-proxy-8996685d-q52jz\" (UID: \"cba183dd-ad5b-469f-9058-67196f9557e5\") " pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz"
Oct 30 13:55:53.122091 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.122000    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba183dd-ad5b-469f-9058-67196f9557e5-tls-certs\") pod \"kube-auth-proxy-8996685d-q52jz\" (UID: \"cba183dd-ad5b-469f-9058-67196f9557e5\") " pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz"
Oct 30 13:55:53.122091 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.122019    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf8w2\" (UniqueName: \"kubernetes.io/projected/cba183dd-ad5b-469f-9058-67196f9557e5-kube-api-access-xf8w2\") pod \"kube-auth-proxy-8996685d-q52jz\" (UID: \"cba183dd-ad5b-469f-9058-67196f9557e5\") " pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz"
Oct 30 13:55:53.132419 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.132402    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\""
Oct 30 13:55:53.132609 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.132593    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-wnsv7\""
Oct 30 13:55:53.222689 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.222662    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cba183dd-ad5b-469f-9058-67196f9557e5-tmp\") pod \"kube-auth-proxy-8996685d-q52jz\" (UID: \"cba183dd-ad5b-469f-9058-67196f9557e5\") " pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz"
Oct 30 13:55:53.223096 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.223078    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba183dd-ad5b-469f-9058-67196f9557e5-tls-certs\") pod \"kube-auth-proxy-8996685d-q52jz\" (UID: \"cba183dd-ad5b-469f-9058-67196f9557e5\") " pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz"
Oct 30 13:55:53.223236 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.223208    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sslw\" (UniqueName: \"kubernetes.io/projected/21ca8cc1-6a8a-4880-849b-a5534fc4ead1-kube-api-access-6sslw\") pod \"odh-observability-584f99c788-k44b2\" (UID: \"21ca8cc1-6a8a-4880-849b-a5534fc4ead1\") " pod="opendatahub/odh-observability-584f99c788-k44b2"
Oct 30 13:55:53.223355 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.223250    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf8w2\" (UniqueName: \"kubernetes.io/projected/cba183dd-ad5b-469f-9058-67196f9557e5-kube-api-access-xf8w2\") pod \"kube-auth-proxy-8996685d-q52jz\" (UID: \"cba183dd-ad5b-469f-9058-67196f9557e5\") " pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz"
Oct 30 13:55:53.223421 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.223392    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/21ca8cc1-6a8a-4880-849b-a5534fc4ead1-webhook-certs\") pod \"odh-observability-584f99c788-k44b2\" (UID: \"21ca8cc1-6a8a-4880-849b-a5534fc4ead1\") " pod="opendatahub/odh-observability-584f99c788-k44b2"
Oct 30 13:55:53.225068 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.225049    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cba183dd-ad5b-469f-9058-67196f9557e5-tmp\") pod \"kube-auth-proxy-8996685d-q52jz\" (UID: \"cba183dd-ad5b-469f-9058-67196f9557e5\") " pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz"
Oct 30 13:55:53.225335 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.225317    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba183dd-ad5b-469f-9058-67196f9557e5-tls-certs\") pod \"kube-auth-proxy-8996685d-q52jz\" (UID: \"cba183dd-ad5b-469f-9058-67196f9557e5\") " pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz"
Oct 30 13:55:53.251047 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.250996    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf8w2\" (UniqueName: \"kubernetes.io/projected/cba183dd-ad5b-469f-9058-67196f9557e5-kube-api-access-xf8w2\") pod \"kube-auth-proxy-8996685d-q52jz\" (UID: \"cba183dd-ad5b-469f-9058-67196f9557e5\") " pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz"
Oct 30 13:55:53.323411 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.323387    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz"
Oct 30 13:55:53.324640 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.324618    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sslw\" (UniqueName: \"kubernetes.io/projected/21ca8cc1-6a8a-4880-849b-a5534fc4ead1-kube-api-access-6sslw\") pod \"odh-observability-584f99c788-k44b2\" (UID: \"21ca8cc1-6a8a-4880-849b-a5534fc4ead1\") " pod="opendatahub/odh-observability-584f99c788-k44b2"
Oct 30 13:55:53.324732 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.324671    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/21ca8cc1-6a8a-4880-849b-a5534fc4ead1-webhook-certs\") pod \"odh-observability-584f99c788-k44b2\" (UID: \"21ca8cc1-6a8a-4880-849b-a5534fc4ead1\") " pod="opendatahub/odh-observability-584f99c788-k44b2"
Oct 30 13:55:53.324827 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:55:53.324802    4530 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 13:55:53.324900 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:55:53.324889    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ca8cc1-6a8a-4880-849b-a5534fc4ead1-webhook-certs podName:21ca8cc1-6a8a-4880-849b-a5534fc4ead1 nodeName:}" failed. No retries permitted until 2026-10-30 13:55:53.82486854 +0000 UTC m=+418.856767471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/21ca8cc1-6a8a-4880-849b-a5534fc4ead1-webhook-certs") pod "odh-observability-584f99c788-k44b2" (UID: "21ca8cc1-6a8a-4880-849b-a5534fc4ead1") : secret "odh-observability-webhook-cert" not found
Oct 30 13:55:53.353107 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.353085    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sslw\" (UniqueName: \"kubernetes.io/projected/21ca8cc1-6a8a-4880-849b-a5534fc4ead1-kube-api-access-6sslw\") pod \"odh-observability-584f99c788-k44b2\" (UID: \"21ca8cc1-6a8a-4880-849b-a5534fc4ead1\") " pod="opendatahub/odh-observability-584f99c788-k44b2"
Oct 30 13:55:53.632068 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.632038    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8996685d-q52jz"]
Oct 30 13:55:53.632708 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:55:53.632683    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba183dd_ad5b_469f_9058_67196f9557e5.slice/crio-9b79841fab19abbd786c7cc42bdafcf4001e361804a2156cebaf64dd50325a22 WatchSource:0}: Error finding container 9b79841fab19abbd786c7cc42bdafcf4001e361804a2156cebaf64dd50325a22: Status 404 returned error can't find the container with id 9b79841fab19abbd786c7cc42bdafcf4001e361804a2156cebaf64dd50325a22
Oct 30 13:55:53.735031 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.735002    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz" event={"ID":"cba183dd-ad5b-469f-9058-67196f9557e5","Type":"ContainerStarted","Data":"9b79841fab19abbd786c7cc42bdafcf4001e361804a2156cebaf64dd50325a22"}
Oct 30 13:55:53.829008 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.828982    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/21ca8cc1-6a8a-4880-849b-a5534fc4ead1-webhook-certs\") pod \"odh-observability-584f99c788-k44b2\" (UID: \"21ca8cc1-6a8a-4880-849b-a5534fc4ead1\") " pod="opendatahub/odh-observability-584f99c788-k44b2"
Oct 30 13:55:53.831334 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.831312    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/21ca8cc1-6a8a-4880-849b-a5534fc4ead1-webhook-certs\") pod \"odh-observability-584f99c788-k44b2\" (UID: \"21ca8cc1-6a8a-4880-849b-a5534fc4ead1\") " pod="opendatahub/odh-observability-584f99c788-k44b2"
Oct 30 13:55:53.975402 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:53.975331    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-k44b2"
Oct 30 13:55:54.274599 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.274574    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-k44b2"]
Oct 30 13:55:54.277000 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:55:54.276971    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ca8cc1_6a8a_4880_849b_a5534fc4ead1.slice/crio-ebf1c61438181564a0be5929b5861b7b95571ecdfaa8f3aa2ae3be205585ee0a WatchSource:0}: Error finding container ebf1c61438181564a0be5929b5861b7b95571ecdfaa8f3aa2ae3be205585ee0a: Status 404 returned error can't find the container with id ebf1c61438181564a0be5929b5861b7b95571ecdfaa8f3aa2ae3be205585ee0a
Oct 30 13:55:54.557118 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.557082    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"]
Oct 30 13:55:54.559626 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.559603    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.578397 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.578148    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 13:55:54.578397 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.578220    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 13:55:54.578397 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.578366    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zzhpw\""
Oct 30 13:55:54.584459 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.584435    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"]
Oct 30 13:55:54.592678 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.592655    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 13:55:54.678661 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.678634    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 13:55:54.737392 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.737361    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.737568 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.737397    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.737568 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.737433    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.737659 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.737643    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.737707 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.737690    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqwzn\" (UniqueName: \"kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-kube-api-access-bqwzn\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.737751 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.737726    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.737859 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.737838    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.740503 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.740478    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-k44b2" event={"ID":"21ca8cc1-6a8a-4880-849b-a5534fc4ead1","Type":"ContainerStarted","Data":"ebf1c61438181564a0be5929b5861b7b95571ecdfaa8f3aa2ae3be205585ee0a"}
Oct 30 13:55:54.839112 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.839039    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.839112 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.839104    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.839330 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.839133    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.839330 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.839273    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.839440 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.839345    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.839440 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.839376    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqwzn\" (UniqueName: \"kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-kube-api-access-bqwzn\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.839440 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.839410    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.839862 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.839836    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.842042 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.842021    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.842171 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.842062    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.842237 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.842175    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.843288 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.843263    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.865794 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.865767    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.866063 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.866044    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqwzn\" (UniqueName: \"kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-kube-api-access-bqwzn\") pod \"istiod-openshift-gateway-7876bfcb74-n49t2\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:54.873825 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:54.873801    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:55.198251 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:55.198227    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"]
Oct 30 13:55:55.198763 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:55:55.198734    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31a0e87_d3a6_472d_a7c9_805ae0c947cd.slice/crio-ac6624f342733f80a8c2ddfd89d90524fc6fbbfdb52055868fe5aaa3c37388b9 WatchSource:0}: Error finding container ac6624f342733f80a8c2ddfd89d90524fc6fbbfdb52055868fe5aaa3c37388b9: Status 404 returned error can't find the container with id ac6624f342733f80a8c2ddfd89d90524fc6fbbfdb52055868fe5aaa3c37388b9
Oct 30 13:55:55.751822 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:55.751726    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2" event={"ID":"e31a0e87-d3a6-472d-a7c9-805ae0c947cd","Type":"ContainerStarted","Data":"ac6624f342733f80a8c2ddfd89d90524fc6fbbfdb52055868fe5aaa3c37388b9"}
Oct 30 13:55:56.285990 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:56.285963    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 13:55:56.759073 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:56.759037    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz" event={"ID":"cba183dd-ad5b-469f-9058-67196f9557e5","Type":"ContainerStarted","Data":"be133290bf3c738db4e2b569ca0ca6d16f1177e2aa5224cb179dd9e3de681f62"}
Oct 30 13:55:56.810461 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:56.810403    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8996685d-q52jz" podStartSLOduration=2.175386612 podStartE2EDuration="4.810385172s" totalImagesPullingTime="2.63499856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:56.808646626 +0000 UTC m=+421.840545566" watchObservedRunningTime="2026-10-30 13:55:56.810385172 +0000 UTC m=+421.842284114"
Oct 30 13:55:57.699593 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:57.699558    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:55:57.699691 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:57.699623    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:55:57.765316 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:57.765288    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2" event={"ID":"e31a0e87-d3a6-472d-a7c9-805ae0c947cd","Type":"ContainerStarted","Data":"f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e"}
Oct 30 13:55:57.765658 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:57.765353    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:55:57.766887 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:57.766865    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-k44b2" event={"ID":"21ca8cc1-6a8a-4880-849b-a5534fc4ead1","Type":"ContainerStarted","Data":"35092948d075bbb4906660d339ad1a22244d8e7885693ede9a48aabbe5199c07"}
Oct 30 13:55:57.767042 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:57.767030    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-k44b2"
Oct 30 13:55:57.918336 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:57.918283    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-k44b2" podStartSLOduration=2.108148637 podStartE2EDuration="4.91826988s" totalImagesPullingTime="2.810121243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:57.91778914 +0000 UTC m=+422.949688079" watchObservedRunningTime="2026-10-30 13:55:57.91826988 +0000 UTC m=+422.950168819"
Oct 30 13:55:57.918578 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:57.918531    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2" podStartSLOduration=1.422249519 podStartE2EDuration="3.918521741s" totalImagesPullingTime="2.496272222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:57.856570685 +0000 UTC m=+422.888469621" watchObservedRunningTime="2026-10-30 13:55:57.918521741 +0000 UTC m=+422.950420680"
Oct 30 13:55:58.773677 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:58.773633    4530 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-n49t2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:55:58.774043 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:55:58.773707    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2" podUID="e31a0e87-d3a6-472d-a7c9-805ae0c947cd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:56:01.772771 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:01.772729    4530 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-n49t2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:56:01.773253 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:01.772780    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2" podUID="e31a0e87-d3a6-472d-a7c9-805ae0c947cd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:56:04.772513 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:04.772478    4530 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-n49t2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:56:04.772920 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:04.772531    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2" podUID="e31a0e87-d3a6-472d-a7c9-805ae0c947cd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:56:07.773233 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:07.773193    4530 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-n49t2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:56:07.773628 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:07.773258    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2" podUID="e31a0e87-d3a6-472d-a7c9-805ae0c947cd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:56:08.774213 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:08.774180    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-k44b2"
Oct 30 13:56:10.772625 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:10.772593    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:56:11.364848 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:56:11.364812    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:56:41.364995 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:56:41.364955    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:56:53.283608 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.283576    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"]
Oct 30 13:56:53.285879 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.285859    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.321882 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.321852    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"]
Oct 30 13:56:53.365795 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.365767    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e761f65-785b-45e0-90df-5ea5697d281b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.365919 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.365796    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2e761f65-785b-45e0-90df-5ea5697d281b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.365919 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.365827    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2e761f65-785b-45e0-90df-5ea5697d281b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.365992 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.365936    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e761f65-785b-45e0-90df-5ea5697d281b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.365992 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.365963    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2e761f65-785b-45e0-90df-5ea5697d281b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.365992 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.365979    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e761f65-785b-45e0-90df-5ea5697d281b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.366118 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.366016    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br872\" (UniqueName: \"kubernetes.io/projected/2e761f65-785b-45e0-90df-5ea5697d281b-kube-api-access-br872\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.466503 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.466475    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2e761f65-785b-45e0-90df-5ea5697d281b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.466636 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.466518    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e761f65-785b-45e0-90df-5ea5697d281b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.466636 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.466535    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2e761f65-785b-45e0-90df-5ea5697d281b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.466709 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.466662    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e761f65-785b-45e0-90df-5ea5697d281b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.466752 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.466712    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br872\" (UniqueName: \"kubernetes.io/projected/2e761f65-785b-45e0-90df-5ea5697d281b-kube-api-access-br872\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.466934 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.466800    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e761f65-785b-45e0-90df-5ea5697d281b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.466934 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.466833    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2e761f65-785b-45e0-90df-5ea5697d281b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.467596 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.467573    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2e761f65-785b-45e0-90df-5ea5697d281b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.468990 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.468966    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2e761f65-785b-45e0-90df-5ea5697d281b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.469091 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.469070    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e761f65-785b-45e0-90df-5ea5697d281b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.469279 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.469259    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e761f65-785b-45e0-90df-5ea5697d281b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.469352 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.469295    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2e761f65-785b-45e0-90df-5ea5697d281b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.493227 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.493200    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br872\" (UniqueName: \"kubernetes.io/projected/2e761f65-785b-45e0-90df-5ea5697d281b-kube-api-access-br872\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.493336 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.493314    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e761f65-785b-45e0-90df-5ea5697d281b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hdsbg\" (UID: \"2e761f65-785b-45e0-90df-5ea5697d281b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.595887 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.595868    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:53.905843 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.905805    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"]
Oct 30 13:56:53.908683 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:56:53.908657    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e761f65_785b_45e0_90df_5ea5697d281b.slice/crio-e47167665ca4a58af5c5ba601b08345995ba1e68e9ab87936bfebbfe434a0f23 WatchSource:0}: Error finding container e47167665ca4a58af5c5ba601b08345995ba1e68e9ab87936bfebbfe434a0f23: Status 404 returned error can't find the container with id e47167665ca4a58af5c5ba601b08345995ba1e68e9ab87936bfebbfe434a0f23
Oct 30 13:56:53.910680 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.910638    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:56:53.910772 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:53.910731    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:56:54.012077 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:54.012038    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg" event={"ID":"2e761f65-785b-45e0-90df-5ea5697d281b","Type":"ContainerStarted","Data":"c6e1b03a35d39ff63cb154ed46e1df35b101cf03d2a30dccbcfeeba2ac04dbba"}
Oct 30 13:56:54.012288 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:54.012094    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg" event={"ID":"2e761f65-785b-45e0-90df-5ea5697d281b","Type":"ContainerStarted","Data":"e47167665ca4a58af5c5ba601b08345995ba1e68e9ab87936bfebbfe434a0f23"}
Oct 30 13:56:54.012376 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:54.012356    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:54.016006 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:54.015979    4530 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hdsbg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:8080/ready\": dial tcp 10.133.0.33:8080: connect: connection refused" start-of-body=
Oct 30 13:56:54.016147 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:54.016031    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg" podUID="2e761f65-785b-45e0-90df-5ea5697d281b" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.33:8080/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Oct 30 13:56:54.070802 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:54.070751    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg" podStartSLOduration=1.070735915 podStartE2EDuration="1.070735915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:54.069853957 +0000 UTC m=+479.101752917" watchObservedRunningTime="2026-10-30 13:56:54.070735915 +0000 UTC m=+479.102634853"
Oct 30 13:56:55.017474 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.017446    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hdsbg"
Oct 30 13:56:55.113669 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.113528    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"]
Oct 30 13:56:55.113838 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.113719    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2" podUID="e31a0e87-d3a6-472d-a7c9-805ae0c947cd" containerName="discovery" containerID="cri-o://f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e" gracePeriod=30
Oct 30 13:56:55.354736 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.354711    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:56:55.487489 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.487456    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-token\" (UniqueName: \"kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-token\") pod \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") "
Oct 30 13:56:55.487489 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.487491    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-cacerts\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-cacerts\") pod \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") "
Oct 30 13:56:55.487731 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.487522    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-ca-configmap\") pod \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") "
Oct 30 13:56:55.487731 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.487552    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-dns-cert\") pod \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") "
Oct 30 13:56:55.487731 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.487592    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-local-certs\") pod \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") "
Oct 30 13:56:55.487731 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.487632    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-kube-api-access-bqwzn\" (UniqueName: \"kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-kube-api-access-bqwzn\") pod \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") "
Oct 30 13:56:55.487731 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.487677    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-kubeconfig\") pod \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\" (UID: \"e31a0e87-d3a6-472d-a7c9-805ae0c947cd\") "
Oct 30 13:56:55.488011 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.487981    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-ca-configmap" pod "e31a0e87-d3a6-472d-a7c9-805ae0c947cd" (UID: "e31a0e87-d3a6-472d-a7c9-805ae0c947cd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:55.489994 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.489966    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-token" pod "e31a0e87-d3a6-472d-a7c9-805ae0c947cd" (UID: "e31a0e87-d3a6-472d-a7c9-805ae0c947cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:55.490171 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.490142    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-local-certs" pod "e31a0e87-d3a6-472d-a7c9-805ae0c947cd" (UID: "e31a0e87-d3a6-472d-a7c9-805ae0c947cd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:55.490413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.490385    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-kube-api-access-bqwzn" pod "e31a0e87-d3a6-472d-a7c9-805ae0c947cd" (UID: "e31a0e87-d3a6-472d-a7c9-805ae0c947cd"). InnerVolumeSpecName "kube-api-access-bqwzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:55.490413 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.490398    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-kubeconfig" pod "e31a0e87-d3a6-472d-a7c9-805ae0c947cd" (UID: "e31a0e87-d3a6-472d-a7c9-805ae0c947cd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:55.490526 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.490418    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-dns-cert" pod "e31a0e87-d3a6-472d-a7c9-805ae0c947cd" (UID: "e31a0e87-d3a6-472d-a7c9-805ae0c947cd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:55.490526 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.490445    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-cacerts" pod "e31a0e87-d3a6-472d-a7c9-805ae0c947cd" (UID: "e31a0e87-d3a6-472d-a7c9-805ae0c947cd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:55.588933 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.588909    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:56:55.588933 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.588931    4530 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-cacerts\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:56:55.589064 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.588941    4530 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-ca-configmap\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:56:55.589064 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.588950    4530 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-csr-dns-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:56:55.589064 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.588959    4530 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-local-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:56:55.589064 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.588967    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqwzn\" (UniqueName: \"kubernetes.io/projected/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-kube-api-access-bqwzn\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:56:55.589064 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:55.588974    4530 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e31a0e87-d3a6-472d-a7c9-805ae0c947cd-istio-kubeconfig\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:56:56.022025 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:56.021943    4530 generic.go:356] "Generic (PLEG): container finished" podID="e31a0e87-d3a6-472d-a7c9-805ae0c947cd" containerID="f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e" exitCode=0
Oct 30 13:56:56.022025 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:56.021988    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"
Oct 30 13:56:56.022521 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:56.022031    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2" event={"ID":"e31a0e87-d3a6-472d-a7c9-805ae0c947cd","Type":"ContainerDied","Data":"f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e"}
Oct 30 13:56:56.022521 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:56.022098    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2" event={"ID":"e31a0e87-d3a6-472d-a7c9-805ae0c947cd","Type":"ContainerDied","Data":"ac6624f342733f80a8c2ddfd89d90524fc6fbbfdb52055868fe5aaa3c37388b9"}
Oct 30 13:56:56.022521 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:56.022120    4530 scope.go:122] "RemoveContainer" containerID="f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e"
Oct 30 13:56:56.031493 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:56.031478    4530 scope.go:122] "RemoveContainer" containerID="f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e"
Oct 30 13:56:56.031782 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:56:56.031766    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e\": container with ID starting with f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e not found: ID does not exist" containerID="f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e"
Oct 30 13:56:56.031845 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:56.031790    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e"} err="failed to get container status \"f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e\": rpc error: code = NotFound desc = could not find container \"f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e\": container with ID starting with f2a9b929f22a6658dbdc9055ad22244d689b15efe3db87c39af5a1adf0bcf89e not found: ID does not exist"
Oct 30 13:56:56.087288 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:56.087266    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"]
Oct 30 13:56:56.101467 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:56.101443    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-n49t2"]
Oct 30 13:56:57.112835 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:56:57.112805    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31a0e87-d3a6-472d-a7c9-805ae0c947cd" path="/var/lib/kubelet/pods/e31a0e87-d3a6-472d-a7c9-805ae0c947cd/volumes"
Oct 30 13:57:11.364798 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:57:11.364704    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:57:41.365024 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:57:41.364988    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:57:55.383693 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.383664    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"]
Oct 30 13:57:55.384034 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.383959    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e31a0e87-d3a6-472d-a7c9-805ae0c947cd" containerName="discovery"
Oct 30 13:57:55.384034 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.383970    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31a0e87-d3a6-472d-a7c9-805ae0c947cd" containerName="discovery"
Oct 30 13:57:55.384034 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.384022    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31a0e87-d3a6-472d-a7c9-805ae0c947cd" containerName="discovery"
Oct 30 13:57:55.388076 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.388055    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:57:55.406307 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.406249    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"]
Oct 30 13:57:55.418907 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.418885    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-bfjd2\""
Oct 30 13:57:55.419011 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.418947    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 13:57:55.421144 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.421101    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 13:57:55.425424 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.425402    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw\" (UID: \"80d72e3d-311d-4490-8633-2864f3987387\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:57:55.425504 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.425450    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwtgx\" (UniqueName: \"kubernetes.io/projected/80d72e3d-311d-4490-8633-2864f3987387-kube-api-access-lwtgx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw\" (UID: \"80d72e3d-311d-4490-8633-2864f3987387\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:57:55.425587 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.425572    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw\" (UID: \"80d72e3d-311d-4490-8633-2864f3987387\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:57:55.481267 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.481225    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"]
Oct 30 13:57:55.484609 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.484588    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:57:55.504833 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.504809    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"]
Oct 30 13:57:55.508101 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.508081    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-f9chw\""
Oct 30 13:57:55.526099 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.526071    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw\" (UID: \"80d72e3d-311d-4490-8633-2864f3987387\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:57:55.526203 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.526105    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz\" (UID: \"66337402-4e49-44d5-a645-d6f65434e824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:57:55.526203 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.526140    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvckr\" (UniqueName: \"kubernetes.io/projected/66337402-4e49-44d5-a645-d6f65434e824-kube-api-access-mvckr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz\" (UID: \"66337402-4e49-44d5-a645-d6f65434e824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:57:55.526318 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.526229    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwtgx\" (UniqueName: \"kubernetes.io/projected/80d72e3d-311d-4490-8633-2864f3987387-kube-api-access-lwtgx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw\" (UID: \"80d72e3d-311d-4490-8633-2864f3987387\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:57:55.526318 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.526286    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz\" (UID: \"66337402-4e49-44d5-a645-d6f65434e824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:57:55.526406 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.526391    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw\" (UID: \"80d72e3d-311d-4490-8633-2864f3987387\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:57:55.526440 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.526409    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw\" (UID: \"80d72e3d-311d-4490-8633-2864f3987387\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:57:55.526635 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.526618    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw\" (UID: \"80d72e3d-311d-4490-8633-2864f3987387\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:57:55.554648 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.554624    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwtgx\" (UniqueName: \"kubernetes.io/projected/80d72e3d-311d-4490-8633-2864f3987387-kube-api-access-lwtgx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw\" (UID: \"80d72e3d-311d-4490-8633-2864f3987387\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:57:55.571373 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.571352    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"]
Oct 30 13:57:55.574929 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.574913    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:57:55.600220 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.600189    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"]
Oct 30 13:57:55.608914 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.608890    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-52vjg\""
Oct 30 13:57:55.626720 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.626692    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2j74\" (UniqueName: \"kubernetes.io/projected/332a7113-4b99-4e44-8749-d9201567c64f-kube-api-access-n2j74\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw\" (UID: \"332a7113-4b99-4e44-8749-d9201567c64f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:57:55.626806 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.626737    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz\" (UID: \"66337402-4e49-44d5-a645-d6f65434e824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:57:55.626845 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.626818    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw\" (UID: \"332a7113-4b99-4e44-8749-d9201567c64f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:57:55.626879 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.626855    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz\" (UID: \"66337402-4e49-44d5-a645-d6f65434e824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:57:55.626879 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.626874    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw\" (UID: \"332a7113-4b99-4e44-8749-d9201567c64f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:57:55.626969 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.626948    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvckr\" (UniqueName: \"kubernetes.io/projected/66337402-4e49-44d5-a645-d6f65434e824-kube-api-access-mvckr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz\" (UID: \"66337402-4e49-44d5-a645-d6f65434e824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:57:55.627105 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.627087    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz\" (UID: \"66337402-4e49-44d5-a645-d6f65434e824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:57:55.627193 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.627175    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz\" (UID: \"66337402-4e49-44d5-a645-d6f65434e824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:57:55.654139 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.654110    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvckr\" (UniqueName: \"kubernetes.io/projected/66337402-4e49-44d5-a645-d6f65434e824-kube-api-access-mvckr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz\" (UID: \"66337402-4e49-44d5-a645-d6f65434e824\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:57:55.683999 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.683941    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"]
Oct 30 13:57:55.687428 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.687411    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:57:55.703832 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.703814    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:57:55.708330 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.708310    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"]
Oct 30 13:57:55.708994 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.708976    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-7m4l2\""
Oct 30 13:57:55.727440 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.727415    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw\" (UID: \"332a7113-4b99-4e44-8749-d9201567c64f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:57:55.727573 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.727470    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65\" (UID: \"523ece42-09fe-43e8-9da8-bb2d67697871\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:57:55.727573 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.727498    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw\" (UID: \"332a7113-4b99-4e44-8749-d9201567c64f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:57:55.727573 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.727525    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmtvs\" (UniqueName: \"kubernetes.io/projected/523ece42-09fe-43e8-9da8-bb2d67697871-kube-api-access-nmtvs\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65\" (UID: \"523ece42-09fe-43e8-9da8-bb2d67697871\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:57:55.727752 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.727707    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65\" (UID: \"523ece42-09fe-43e8-9da8-bb2d67697871\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:57:55.727811 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.727764    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2j74\" (UniqueName: \"kubernetes.io/projected/332a7113-4b99-4e44-8749-d9201567c64f-kube-api-access-n2j74\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw\" (UID: \"332a7113-4b99-4e44-8749-d9201567c64f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:57:55.727990 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.727962    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw\" (UID: \"332a7113-4b99-4e44-8749-d9201567c64f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:57:55.728050 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.728006    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw\" (UID: \"332a7113-4b99-4e44-8749-d9201567c64f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:57:55.756283 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.756253    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2j74\" (UniqueName: \"kubernetes.io/projected/332a7113-4b99-4e44-8749-d9201567c64f-kube-api-access-n2j74\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw\" (UID: \"332a7113-4b99-4e44-8749-d9201567c64f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:57:55.800902 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.800873    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:57:55.829018 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.828985    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65\" (UID: \"523ece42-09fe-43e8-9da8-bb2d67697871\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:57:55.829126 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.829023    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtvs\" (UniqueName: \"kubernetes.io/projected/523ece42-09fe-43e8-9da8-bb2d67697871-kube-api-access-nmtvs\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65\" (UID: \"523ece42-09fe-43e8-9da8-bb2d67697871\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:57:55.829126 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.829083    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65\" (UID: \"523ece42-09fe-43e8-9da8-bb2d67697871\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:57:55.829420 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.829404    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65\" (UID: \"523ece42-09fe-43e8-9da8-bb2d67697871\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:57:55.829470 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.829453    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65\" (UID: \"523ece42-09fe-43e8-9da8-bb2d67697871\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:57:55.858195 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.858168    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmtvs\" (UniqueName: \"kubernetes.io/projected/523ece42-09fe-43e8-9da8-bb2d67697871-kube-api-access-nmtvs\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65\" (UID: \"523ece42-09fe-43e8-9da8-bb2d67697871\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:57:55.891009 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:55.890983    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:57:56.002781 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.002750    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:57:56.003167 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:57:56.003141    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d72e3d_311d_4490_8633_2864f3987387.slice/crio-15518cef571942864b92b5de70f96fba00ceec9572d5e1b15e57cb1fdc074604 WatchSource:0}: Error finding container 15518cef571942864b92b5de70f96fba00ceec9572d5e1b15e57cb1fdc074604: Status 404 returned error can't find the container with id 15518cef571942864b92b5de70f96fba00ceec9572d5e1b15e57cb1fdc074604
Oct 30 13:57:56.003560 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.003524    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"]
Oct 30 13:57:56.099659 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.099632    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"]
Oct 30 13:57:56.100097 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:57:56.100062    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66337402_4e49_44d5_a645_d6f65434e824.slice/crio-5506cf8d4da6a03935bcb91b26696e80434bc1e3291ee3f8bedfc0fdc2ebcee7 WatchSource:0}: Error finding container 5506cf8d4da6a03935bcb91b26696e80434bc1e3291ee3f8bedfc0fdc2ebcee7: Status 404 returned error can't find the container with id 5506cf8d4da6a03935bcb91b26696e80434bc1e3291ee3f8bedfc0fdc2ebcee7
Oct 30 13:57:56.190793 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.190769    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"]
Oct 30 13:57:56.191089 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:57:56.191065    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332a7113_4b99_4e44_8749_d9201567c64f.slice/crio-6fb60f0bb1edde738fd2de9f69da16fefec7251012d76022e35bc831b6db1ab9 WatchSource:0}: Error finding container 6fb60f0bb1edde738fd2de9f69da16fefec7251012d76022e35bc831b6db1ab9: Status 404 returned error can't find the container with id 6fb60f0bb1edde738fd2de9f69da16fefec7251012d76022e35bc831b6db1ab9
Oct 30 13:57:56.275367 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.275334    4530 generic.go:356] "Generic (PLEG): container finished" podID="66337402-4e49-44d5-a645-d6f65434e824" containerID="fa0f196825089117872efa132b99a4e49922f119cd1888a967b8609adff7f093" exitCode=0
Oct 30 13:57:56.275516 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.275395    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz" event={"ID":"66337402-4e49-44d5-a645-d6f65434e824","Type":"ContainerDied","Data":"fa0f196825089117872efa132b99a4e49922f119cd1888a967b8609adff7f093"}
Oct 30 13:57:56.275516 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.275453    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz" event={"ID":"66337402-4e49-44d5-a645-d6f65434e824","Type":"ContainerStarted","Data":"5506cf8d4da6a03935bcb91b26696e80434bc1e3291ee3f8bedfc0fdc2ebcee7"}
Oct 30 13:57:56.277178 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.277157    4530 generic.go:356] "Generic (PLEG): container finished" podID="80d72e3d-311d-4490-8633-2864f3987387" containerID="15617a0a9c4734a467a195ce10180f2255d4b49b2fb0061de7b6d0fd83f52087" exitCode=0
Oct 30 13:57:56.277291 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.277225    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw" event={"ID":"80d72e3d-311d-4490-8633-2864f3987387","Type":"ContainerDied","Data":"15617a0a9c4734a467a195ce10180f2255d4b49b2fb0061de7b6d0fd83f52087"}
Oct 30 13:57:56.277291 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.277275    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw" event={"ID":"80d72e3d-311d-4490-8633-2864f3987387","Type":"ContainerStarted","Data":"15518cef571942864b92b5de70f96fba00ceec9572d5e1b15e57cb1fdc074604"}
Oct 30 13:57:56.279026 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.279002    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw" event={"ID":"332a7113-4b99-4e44-8749-d9201567c64f","Type":"ContainerStarted","Data":"a5a906c86c163843bf4ab9a67f583d6c8ce62d4ffe0fc81f057f1884ef325835"}
Oct 30 13:57:56.279134 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.279049    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw" event={"ID":"332a7113-4b99-4e44-8749-d9201567c64f","Type":"ContainerStarted","Data":"6fb60f0bb1edde738fd2de9f69da16fefec7251012d76022e35bc831b6db1ab9"}
Oct 30 13:57:56.308076 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:56.308055    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"]
Oct 30 13:57:56.366272 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:57:56.366247    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523ece42_09fe_43e8_9da8_bb2d67697871.slice/crio-2b16070a9dfcbbcef861fb04e3a9913cedf8cb6371465d679d1993a7d6ad6ad6 WatchSource:0}: Error finding container 2b16070a9dfcbbcef861fb04e3a9913cedf8cb6371465d679d1993a7d6ad6ad6: Status 404 returned error can't find the container with id 2b16070a9dfcbbcef861fb04e3a9913cedf8cb6371465d679d1993a7d6ad6ad6
Oct 30 13:57:57.284775 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:57.284717    4530 generic.go:356] "Generic (PLEG): container finished" podID="80d72e3d-311d-4490-8633-2864f3987387" containerID="69738bd6dd41636dcd61b3a7c1e0c40b0ee26fff33c171e83b758853373c3d25" exitCode=0
Oct 30 13:57:57.285126 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:57.284802    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw" event={"ID":"80d72e3d-311d-4490-8633-2864f3987387","Type":"ContainerDied","Data":"69738bd6dd41636dcd61b3a7c1e0c40b0ee26fff33c171e83b758853373c3d25"}
Oct 30 13:57:57.286474 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:57.286449    4530 generic.go:356] "Generic (PLEG): container finished" podID="332a7113-4b99-4e44-8749-d9201567c64f" containerID="a5a906c86c163843bf4ab9a67f583d6c8ce62d4ffe0fc81f057f1884ef325835" exitCode=0
Oct 30 13:57:57.286570 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:57.286521    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw" event={"ID":"332a7113-4b99-4e44-8749-d9201567c64f","Type":"ContainerDied","Data":"a5a906c86c163843bf4ab9a67f583d6c8ce62d4ffe0fc81f057f1884ef325835"}
Oct 30 13:57:57.288749 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:57.288617    4530 generic.go:356] "Generic (PLEG): container finished" podID="66337402-4e49-44d5-a645-d6f65434e824" containerID="88ce82938c3b5a4f9ba59ba0b752c4d56c3a6db5949d53aac887f072f8c973fc" exitCode=0
Oct 30 13:57:57.288749 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:57.288655    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz" event={"ID":"66337402-4e49-44d5-a645-d6f65434e824","Type":"ContainerDied","Data":"88ce82938c3b5a4f9ba59ba0b752c4d56c3a6db5949d53aac887f072f8c973fc"}
Oct 30 13:57:57.290334 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:57.290314    4530 generic.go:356] "Generic (PLEG): container finished" podID="523ece42-09fe-43e8-9da8-bb2d67697871" containerID="823314ec73c6aa3dcb0bde0aa156756a999a1b982676637b9a528444988ff468" exitCode=0
Oct 30 13:57:57.290412 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:57.290345    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65" event={"ID":"523ece42-09fe-43e8-9da8-bb2d67697871","Type":"ContainerDied","Data":"823314ec73c6aa3dcb0bde0aa156756a999a1b982676637b9a528444988ff468"}
Oct 30 13:57:57.290412 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:57.290375    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65" event={"ID":"523ece42-09fe-43e8-9da8-bb2d67697871","Type":"ContainerStarted","Data":"2b16070a9dfcbbcef861fb04e3a9913cedf8cb6371465d679d1993a7d6ad6ad6"}
Oct 30 13:57:58.297114 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:58.297087    4530 generic.go:356] "Generic (PLEG): container finished" podID="66337402-4e49-44d5-a645-d6f65434e824" containerID="9f55041e7caea8ddfb3ffc17881af96ac489dd822eb9710dff6c7a43d0a87b7c" exitCode=0
Oct 30 13:57:58.297476 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:58.297170    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz" event={"ID":"66337402-4e49-44d5-a645-d6f65434e824","Type":"ContainerDied","Data":"9f55041e7caea8ddfb3ffc17881af96ac489dd822eb9710dff6c7a43d0a87b7c"}
Oct 30 13:57:58.298951 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:58.298930    4530 generic.go:356] "Generic (PLEG): container finished" podID="523ece42-09fe-43e8-9da8-bb2d67697871" containerID="342a7adf15600091b5ff358c6fe307001208de7f905b99d7dd5aa16dcf36581d" exitCode=0
Oct 30 13:57:58.299047 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:58.299011    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65" event={"ID":"523ece42-09fe-43e8-9da8-bb2d67697871","Type":"ContainerDied","Data":"342a7adf15600091b5ff358c6fe307001208de7f905b99d7dd5aa16dcf36581d"}
Oct 30 13:57:58.301528 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:58.301444    4530 generic.go:356] "Generic (PLEG): container finished" podID="80d72e3d-311d-4490-8633-2864f3987387" containerID="e4b12fb15d73504b88edfc81777ea6a0aa076009ba88dee75566e90647f83887" exitCode=0
Oct 30 13:57:58.301528 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:58.301469    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw" event={"ID":"80d72e3d-311d-4490-8633-2864f3987387","Type":"ContainerDied","Data":"e4b12fb15d73504b88edfc81777ea6a0aa076009ba88dee75566e90647f83887"}
Oct 30 13:57:59.307554 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.307515    4530 generic.go:356] "Generic (PLEG): container finished" podID="523ece42-09fe-43e8-9da8-bb2d67697871" containerID="882543441d255729b7266223618722da293f281eca99285f94563a51c80ee023" exitCode=0
Oct 30 13:57:59.307935 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.307593    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65" event={"ID":"523ece42-09fe-43e8-9da8-bb2d67697871","Type":"ContainerDied","Data":"882543441d255729b7266223618722da293f281eca99285f94563a51c80ee023"}
Oct 30 13:57:59.309486 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.309464    4530 generic.go:356] "Generic (PLEG): container finished" podID="332a7113-4b99-4e44-8749-d9201567c64f" containerID="162118adb2e9e84ba0c03a17c867810653cf0afb9455bfe909dab7d611da0400" exitCode=0
Oct 30 13:57:59.309626 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.309552    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw" event={"ID":"332a7113-4b99-4e44-8749-d9201567c64f","Type":"ContainerDied","Data":"162118adb2e9e84ba0c03a17c867810653cf0afb9455bfe909dab7d611da0400"}
Oct 30 13:57:59.448897 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.448875    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:57:59.475300 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.475279    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:57:59.558673 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.558595    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-util\" (UniqueName: \"kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-util\") pod \"66337402-4e49-44d5-a645-d6f65434e824\" (UID: \"66337402-4e49-44d5-a645-d6f65434e824\") "
Oct 30 13:57:59.558673 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.558642    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66337402-4e49-44d5-a645-d6f65434e824-kube-api-access-mvckr\" (UniqueName: \"kubernetes.io/projected/66337402-4e49-44d5-a645-d6f65434e824-kube-api-access-mvckr\") pod \"66337402-4e49-44d5-a645-d6f65434e824\" (UID: \"66337402-4e49-44d5-a645-d6f65434e824\") "
Oct 30 13:57:59.558848 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.558697    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-bundle\" (UniqueName: \"kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-bundle\") pod \"66337402-4e49-44d5-a645-d6f65434e824\" (UID: \"66337402-4e49-44d5-a645-d6f65434e824\") "
Oct 30 13:57:59.558848 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.558720    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-util\" (UniqueName: \"kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-util\") pod \"80d72e3d-311d-4490-8633-2864f3987387\" (UID: \"80d72e3d-311d-4490-8633-2864f3987387\") "
Oct 30 13:57:59.558848 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.558758    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-bundle\" (UniqueName: \"kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-bundle\") pod \"80d72e3d-311d-4490-8633-2864f3987387\" (UID: \"80d72e3d-311d-4490-8633-2864f3987387\") "
Oct 30 13:57:59.558848 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.558804    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d72e3d-311d-4490-8633-2864f3987387-kube-api-access-lwtgx\" (UniqueName: \"kubernetes.io/projected/80d72e3d-311d-4490-8633-2864f3987387-kube-api-access-lwtgx\") pod \"80d72e3d-311d-4490-8633-2864f3987387\" (UID: \"80d72e3d-311d-4490-8633-2864f3987387\") "
Oct 30 13:57:59.559329 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.559303    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-bundle" pod "66337402-4e49-44d5-a645-d6f65434e824" (UID: "66337402-4e49-44d5-a645-d6f65434e824"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:59.560056 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.560034    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-bundle" pod "80d72e3d-311d-4490-8633-2864f3987387" (UID: "80d72e3d-311d-4490-8633-2864f3987387"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:59.561464 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.561441    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d72e3d-311d-4490-8633-2864f3987387-kube-api-access-lwtgx" pod "80d72e3d-311d-4490-8633-2864f3987387" (UID: "80d72e3d-311d-4490-8633-2864f3987387"). InnerVolumeSpecName "kube-api-access-lwtgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:59.561535 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.561472    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66337402-4e49-44d5-a645-d6f65434e824-kube-api-access-mvckr" pod "66337402-4e49-44d5-a645-d6f65434e824" (UID: "66337402-4e49-44d5-a645-d6f65434e824"). InnerVolumeSpecName "kube-api-access-mvckr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:59.566607 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.566580    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-util" pod "66337402-4e49-44d5-a645-d6f65434e824" (UID: "66337402-4e49-44d5-a645-d6f65434e824"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:59.567820 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.567802    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-util" pod "80d72e3d-311d-4490-8633-2864f3987387" (UID: "80d72e3d-311d-4490-8633-2864f3987387"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:59.659967 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.659933    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwtgx\" (UniqueName: \"kubernetes.io/projected/80d72e3d-311d-4490-8633-2864f3987387-kube-api-access-lwtgx\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:57:59.659967 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.659966    4530 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:57:59.659967 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.659976    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvckr\" (UniqueName: \"kubernetes.io/projected/66337402-4e49-44d5-a645-d6f65434e824-kube-api-access-mvckr\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:57:59.660123 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.659985    4530 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66337402-4e49-44d5-a645-d6f65434e824-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:57:59.660123 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.659993    4530 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:57:59.660123 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:57:59.660000    4530 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80d72e3d-311d-4490-8633-2864f3987387-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:58:00.315243 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.315215    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw"
Oct 30 13:58:00.315708 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.315241    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cht5gw" event={"ID":"80d72e3d-311d-4490-8633-2864f3987387","Type":"ContainerDied","Data":"15518cef571942864b92b5de70f96fba00ceec9572d5e1b15e57cb1fdc074604"}
Oct 30 13:58:00.315708 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.315291    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15518cef571942864b92b5de70f96fba00ceec9572d5e1b15e57cb1fdc074604"
Oct 30 13:58:00.317395 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.317372    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw" event={"ID":"332a7113-4b99-4e44-8749-d9201567c64f","Type":"ContainerStarted","Data":"d9c63a52c8bf8a97b2323c6a7983dc37d1555e4f165ddb64dcc57b64732aa825"}
Oct 30 13:58:00.319370 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.319349    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz"
Oct 30 13:58:00.319463 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.319371    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dxvrz" event={"ID":"66337402-4e49-44d5-a645-d6f65434e824","Type":"ContainerDied","Data":"5506cf8d4da6a03935bcb91b26696e80434bc1e3291ee3f8bedfc0fdc2ebcee7"}
Oct 30 13:58:00.319463 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.319412    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5506cf8d4da6a03935bcb91b26696e80434bc1e3291ee3f8bedfc0fdc2ebcee7"
Oct 30 13:58:00.365219 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.365166    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw" podStartSLOduration=4.385101237 podStartE2EDuration="5.365152366s" totalImagesPullingTime="980.051129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:57:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:00.364620862 +0000 UTC m=+545.396519804" watchObservedRunningTime="2026-10-30 13:58:00.365152366 +0000 UTC m=+545.397051305"
Oct 30 13:58:00.447021 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.446993    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:58:00.566765 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.566708    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523ece42-09fe-43e8-9da8-bb2d67697871-kube-api-access-nmtvs\" (UniqueName: \"kubernetes.io/projected/523ece42-09fe-43e8-9da8-bb2d67697871-kube-api-access-nmtvs\") pod \"523ece42-09fe-43e8-9da8-bb2d67697871\" (UID: \"523ece42-09fe-43e8-9da8-bb2d67697871\") "
Oct 30 13:58:00.566765 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.566747    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-bundle\" (UniqueName: \"kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-bundle\") pod \"523ece42-09fe-43e8-9da8-bb2d67697871\" (UID: \"523ece42-09fe-43e8-9da8-bb2d67697871\") "
Oct 30 13:58:00.566928 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.566778    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-util\" (UniqueName: \"kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-util\") pod \"523ece42-09fe-43e8-9da8-bb2d67697871\" (UID: \"523ece42-09fe-43e8-9da8-bb2d67697871\") "
Oct 30 13:58:00.567356 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.567335    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-bundle" pod "523ece42-09fe-43e8-9da8-bb2d67697871" (UID: "523ece42-09fe-43e8-9da8-bb2d67697871"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:00.568891 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.568867    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523ece42-09fe-43e8-9da8-bb2d67697871-kube-api-access-nmtvs" pod "523ece42-09fe-43e8-9da8-bb2d67697871" (UID: "523ece42-09fe-43e8-9da8-bb2d67697871"). InnerVolumeSpecName "kube-api-access-nmtvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:00.573211 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.573184    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-util" pod "523ece42-09fe-43e8-9da8-bb2d67697871" (UID: "523ece42-09fe-43e8-9da8-bb2d67697871"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:00.667809 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.667784    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmtvs\" (UniqueName: \"kubernetes.io/projected/523ece42-09fe-43e8-9da8-bb2d67697871-kube-api-access-nmtvs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:58:00.667938 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.667810    4530 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:58:00.667938 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:00.667825    4530 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/523ece42-09fe-43e8-9da8-bb2d67697871-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:58:01.325270 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:01.325241    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65"
Oct 30 13:58:01.325270 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:01.325251    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atjc65" event={"ID":"523ece42-09fe-43e8-9da8-bb2d67697871","Type":"ContainerDied","Data":"2b16070a9dfcbbcef861fb04e3a9913cedf8cb6371465d679d1993a7d6ad6ad6"}
Oct 30 13:58:01.325742 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:01.325300    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b16070a9dfcbbcef861fb04e3a9913cedf8cb6371465d679d1993a7d6ad6ad6"
Oct 30 13:58:10.366584 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:10.366533    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 13:58:10.367201 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:10.367180    4530 generic.go:356] "Generic (PLEG): container finished" podID="332a7113-4b99-4e44-8749-d9201567c64f" containerID="d9c63a52c8bf8a97b2323c6a7983dc37d1555e4f165ddb64dcc57b64732aa825" exitCode=1
Oct 30 13:58:10.367298 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:10.367262    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw" event={"ID":"332a7113-4b99-4e44-8749-d9201567c64f","Type":"ContainerDied","Data":"d9c63a52c8bf8a97b2323c6a7983dc37d1555e4f165ddb64dcc57b64732aa825"}
Oct 30 13:58:11.365451 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:58:11.365414    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:58:11.496663 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:11.496641    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 13:58:11.499479 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:11.499459    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:58:11.546453 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:11.546431    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332a7113-4b99-4e44-8749-d9201567c64f-kube-api-access-n2j74\" (UniqueName: \"kubernetes.io/projected/332a7113-4b99-4e44-8749-d9201567c64f-kube-api-access-n2j74\") pod \"332a7113-4b99-4e44-8749-d9201567c64f\" (UID: \"332a7113-4b99-4e44-8749-d9201567c64f\") "
Oct 30 13:58:11.546594 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:11.546473    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-util\" (UniqueName: \"kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-util\") pod \"332a7113-4b99-4e44-8749-d9201567c64f\" (UID: \"332a7113-4b99-4e44-8749-d9201567c64f\") "
Oct 30 13:58:11.546594 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:11.546581    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-bundle\") pod \"332a7113-4b99-4e44-8749-d9201567c64f\" (UID: \"332a7113-4b99-4e44-8749-d9201567c64f\") "
Oct 30 13:58:11.547062 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:11.547032    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-bundle" pod "332a7113-4b99-4e44-8749-d9201567c64f" (UID: "332a7113-4b99-4e44-8749-d9201567c64f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:11.551424 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:11.551395    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332a7113-4b99-4e44-8749-d9201567c64f-kube-api-access-n2j74" pod "332a7113-4b99-4e44-8749-d9201567c64f" (UID: "332a7113-4b99-4e44-8749-d9201567c64f"). InnerVolumeSpecName "kube-api-access-n2j74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:11.551685 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:11.551652    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-util" pod "332a7113-4b99-4e44-8749-d9201567c64f" (UID: "332a7113-4b99-4e44-8749-d9201567c64f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:11.647863 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:11.647776    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2j74\" (UniqueName: \"kubernetes.io/projected/332a7113-4b99-4e44-8749-d9201567c64f-kube-api-access-n2j74\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:58:11.647863 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:11.647809    4530 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:58:11.647863 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:11.647819    4530 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/332a7113-4b99-4e44-8749-d9201567c64f-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:58:12.378282 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:12.378253    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 13:58:12.378997 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:12.378975    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw"
Oct 30 13:58:12.379094 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:12.378973    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw" event={"ID":"332a7113-4b99-4e44-8749-d9201567c64f","Type":"ContainerDied","Data":"6fb60f0bb1edde738fd2de9f69da16fefec7251012d76022e35bc831b6db1ab9"}
Oct 30 13:58:12.379130 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:12.379095    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fb60f0bb1edde738fd2de9f69da16fefec7251012d76022e35bc831b6db1ab9"
Oct 30 13:58:19.032015 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.031983    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"]
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032265    4530 cpu_manager.go:414] "containerMap: removing container" podUID="80d72e3d-311d-4490-8633-2864f3987387" containerName="pull"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032275    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d72e3d-311d-4490-8633-2864f3987387" containerName="pull"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032286    4530 cpu_manager.go:414] "containerMap: removing container" podUID="523ece42-09fe-43e8-9da8-bb2d67697871" containerName="pull"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032291    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523ece42-09fe-43e8-9da8-bb2d67697871" containerName="pull"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032300    4530 cpu_manager.go:414] "containerMap: removing container" podUID="523ece42-09fe-43e8-9da8-bb2d67697871" containerName="extract"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032304    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523ece42-09fe-43e8-9da8-bb2d67697871" containerName="extract"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032311    4530 cpu_manager.go:414] "containerMap: removing container" podUID="332a7113-4b99-4e44-8749-d9201567c64f" containerName="pull"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032316    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332a7113-4b99-4e44-8749-d9201567c64f" containerName="pull"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032332    4530 cpu_manager.go:414] "containerMap: removing container" podUID="66337402-4e49-44d5-a645-d6f65434e824" containerName="pull"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032337    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66337402-4e49-44d5-a645-d6f65434e824" containerName="pull"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032343    4530 cpu_manager.go:414] "containerMap: removing container" podUID="332a7113-4b99-4e44-8749-d9201567c64f" containerName="extract"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032348    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332a7113-4b99-4e44-8749-d9201567c64f" containerName="extract"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032355    4530 cpu_manager.go:414] "containerMap: removing container" podUID="66337402-4e49-44d5-a645-d6f65434e824" containerName="util"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032359    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66337402-4e49-44d5-a645-d6f65434e824" containerName="util"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032365    4530 cpu_manager.go:414] "containerMap: removing container" podUID="80d72e3d-311d-4490-8633-2864f3987387" containerName="util"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032370    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d72e3d-311d-4490-8633-2864f3987387" containerName="util"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032375    4530 cpu_manager.go:414] "containerMap: removing container" podUID="332a7113-4b99-4e44-8749-d9201567c64f" containerName="util"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032379    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332a7113-4b99-4e44-8749-d9201567c64f" containerName="util"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032386    4530 cpu_manager.go:414] "containerMap: removing container" podUID="523ece42-09fe-43e8-9da8-bb2d67697871" containerName="util"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032390    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523ece42-09fe-43e8-9da8-bb2d67697871" containerName="util"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032396    4530 cpu_manager.go:414] "containerMap: removing container" podUID="66337402-4e49-44d5-a645-d6f65434e824" containerName="extract"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032400    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66337402-4e49-44d5-a645-d6f65434e824" containerName="extract"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032411    4530 cpu_manager.go:414] "containerMap: removing container" podUID="80d72e3d-311d-4490-8633-2864f3987387" containerName="extract"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032417    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d72e3d-311d-4490-8633-2864f3987387" containerName="extract"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032462    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="523ece42-09fe-43e8-9da8-bb2d67697871" containerName="extract"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032472    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="332a7113-4b99-4e44-8749-d9201567c64f" containerName="extract"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032479    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="66337402-4e49-44d5-a645-d6f65434e824" containerName="extract"
Oct 30 13:58:19.032471 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.032486    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d72e3d-311d-4490-8633-2864f3987387" containerName="extract"
Oct 30 13:58:19.035663 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.035646    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:19.055026 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.054998    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"]
Oct 30 13:58:19.066910 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.066881    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 13:58:19.067195 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.067176    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-52vjg\""
Oct 30 13:58:19.067258 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.067207    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 13:58:19.106916 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.106889    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcwj9\" (UniqueName: \"kubernetes.io/projected/dda65499-54d0-4065-b3c1-d96305448dbb-kube-api-access-hcwj9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g\" (UID: \"dda65499-54d0-4065-b3c1-d96305448dbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:19.107010 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.106986    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g\" (UID: \"dda65499-54d0-4065-b3c1-d96305448dbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:19.107050 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.107012    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g\" (UID: \"dda65499-54d0-4065-b3c1-d96305448dbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:19.208056 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.208031    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g\" (UID: \"dda65499-54d0-4065-b3c1-d96305448dbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:19.208146 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.208062    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g\" (UID: \"dda65499-54d0-4065-b3c1-d96305448dbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:19.208146 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.208107    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcwj9\" (UniqueName: \"kubernetes.io/projected/dda65499-54d0-4065-b3c1-d96305448dbb-kube-api-access-hcwj9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g\" (UID: \"dda65499-54d0-4065-b3c1-d96305448dbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:19.208461 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.208443    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g\" (UID: \"dda65499-54d0-4065-b3c1-d96305448dbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:19.208510 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.208457    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g\" (UID: \"dda65499-54d0-4065-b3c1-d96305448dbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:19.234784 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.234764    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcwj9\" (UniqueName: \"kubernetes.io/projected/dda65499-54d0-4065-b3c1-d96305448dbb-kube-api-access-hcwj9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g\" (UID: \"dda65499-54d0-4065-b3c1-d96305448dbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:19.350419 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.350399    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:19.649299 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:58:19.649265    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda65499_54d0_4065_b3c1_d96305448dbb.slice/crio-fc4c5dd98c7e1bd3fca505868efc1f3589516e9cef5f76abc4c9e9429c1a6860 WatchSource:0}: Error finding container fc4c5dd98c7e1bd3fca505868efc1f3589516e9cef5f76abc4c9e9429c1a6860: Status 404 returned error can't find the container with id fc4c5dd98c7e1bd3fca505868efc1f3589516e9cef5f76abc4c9e9429c1a6860
Oct 30 13:58:19.649734 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:19.649706    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"]
Oct 30 13:58:20.418765 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:20.418728    4530 generic.go:356] "Generic (PLEG): container finished" podID="dda65499-54d0-4065-b3c1-d96305448dbb" containerID="e8b6101264de70ccf84bf87ffb4d3ca2562fdc6a16720a3d6cb8f85a5effc167" exitCode=0
Oct 30 13:58:20.419200 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:20.418788    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g" event={"ID":"dda65499-54d0-4065-b3c1-d96305448dbb","Type":"ContainerDied","Data":"e8b6101264de70ccf84bf87ffb4d3ca2562fdc6a16720a3d6cb8f85a5effc167"}
Oct 30 13:58:20.419200 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:20.418837    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g" event={"ID":"dda65499-54d0-4065-b3c1-d96305448dbb","Type":"ContainerStarted","Data":"fc4c5dd98c7e1bd3fca505868efc1f3589516e9cef5f76abc4c9e9429c1a6860"}
Oct 30 13:58:21.425495 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:21.425458    4530 generic.go:356] "Generic (PLEG): container finished" podID="dda65499-54d0-4065-b3c1-d96305448dbb" containerID="68cdfc2533fa3c25eb01f4edf174fdf48bfa9934284f5257f874e4e63ebf23b6" exitCode=0
Oct 30 13:58:21.425989 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:21.425510    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g" event={"ID":"dda65499-54d0-4065-b3c1-d96305448dbb","Type":"ContainerDied","Data":"68cdfc2533fa3c25eb01f4edf174fdf48bfa9934284f5257f874e4e63ebf23b6"}
Oct 30 13:58:22.432636 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:22.432606    4530 generic.go:356] "Generic (PLEG): container finished" podID="dda65499-54d0-4065-b3c1-d96305448dbb" containerID="f03543e38b675409b81f957d8a86512e241ea2c202369c2b56cb55df5940ace3" exitCode=0
Oct 30 13:58:22.433037 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:22.432689    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g" event={"ID":"dda65499-54d0-4065-b3c1-d96305448dbb","Type":"ContainerDied","Data":"f03543e38b675409b81f957d8a86512e241ea2c202369c2b56cb55df5940ace3"}
Oct 30 13:58:23.557769 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:23.557745    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:23.639608 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:23.639582    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-util\" (UniqueName: \"kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-util\") pod \"dda65499-54d0-4065-b3c1-d96305448dbb\" (UID: \"dda65499-54d0-4065-b3c1-d96305448dbb\") "
Oct 30 13:58:23.639748 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:23.639628    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-bundle\") pod \"dda65499-54d0-4065-b3c1-d96305448dbb\" (UID: \"dda65499-54d0-4065-b3c1-d96305448dbb\") "
Oct 30 13:58:23.639748 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:23.639726    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda65499-54d0-4065-b3c1-d96305448dbb-kube-api-access-hcwj9\" (UniqueName: \"kubernetes.io/projected/dda65499-54d0-4065-b3c1-d96305448dbb-kube-api-access-hcwj9\") pod \"dda65499-54d0-4065-b3c1-d96305448dbb\" (UID: \"dda65499-54d0-4065-b3c1-d96305448dbb\") "
Oct 30 13:58:23.640116 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:23.640089    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-bundle" pod "dda65499-54d0-4065-b3c1-d96305448dbb" (UID: "dda65499-54d0-4065-b3c1-d96305448dbb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:23.641664 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:23.641641    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda65499-54d0-4065-b3c1-d96305448dbb-kube-api-access-hcwj9" pod "dda65499-54d0-4065-b3c1-d96305448dbb" (UID: "dda65499-54d0-4065-b3c1-d96305448dbb"). InnerVolumeSpecName "kube-api-access-hcwj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:23.645774 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:23.645689    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-util" pod "dda65499-54d0-4065-b3c1-d96305448dbb" (UID: "dda65499-54d0-4065-b3c1-d96305448dbb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:23.740849 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:23.740788    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcwj9\" (UniqueName: \"kubernetes.io/projected/dda65499-54d0-4065-b3c1-d96305448dbb-kube-api-access-hcwj9\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:58:23.740849 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:23.740811    4530 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:58:23.740849 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:23.740826    4530 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dda65499-54d0-4065-b3c1-d96305448dbb-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:58:24.444010 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:24.443980    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g" event={"ID":"dda65499-54d0-4065-b3c1-d96305448dbb","Type":"ContainerDied","Data":"fc4c5dd98c7e1bd3fca505868efc1f3589516e9cef5f76abc4c9e9429c1a6860"}
Oct 30 13:58:24.444010 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:24.443997    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjk6g"
Oct 30 13:58:24.444187 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:24.444026    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc4c5dd98c7e1bd3fca505868efc1f3589516e9cef5f76abc4c9e9429c1a6860"
Oct 30 13:58:35.955768 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.955733    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"]
Oct 30 13:58:35.959454 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.956050    4530 cpu_manager.go:414] "containerMap: removing container" podUID="dda65499-54d0-4065-b3c1-d96305448dbb" containerName="util"
Oct 30 13:58:35.959454 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.956063    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda65499-54d0-4065-b3c1-d96305448dbb" containerName="util"
Oct 30 13:58:35.959454 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.956079    4530 cpu_manager.go:414] "containerMap: removing container" podUID="dda65499-54d0-4065-b3c1-d96305448dbb" containerName="pull"
Oct 30 13:58:35.959454 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.956084    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda65499-54d0-4065-b3c1-d96305448dbb" containerName="pull"
Oct 30 13:58:35.959454 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.956098    4530 cpu_manager.go:414] "containerMap: removing container" podUID="dda65499-54d0-4065-b3c1-d96305448dbb" containerName="extract"
Oct 30 13:58:35.959454 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.956103    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda65499-54d0-4065-b3c1-d96305448dbb" containerName="extract"
Oct 30 13:58:35.959454 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.956151    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda65499-54d0-4065-b3c1-d96305448dbb" containerName="extract"
Oct 30 13:58:35.960242 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.960226    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"
Oct 30 13:58:35.973727 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.973701    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"]
Oct 30 13:58:35.985580 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.985554    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 13:58:35.985711 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.985669    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 13:58:35.985767 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:35.985725    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zdx8r\""
Oct 30 13:58:36.131117 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:36.131085    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63628291-cdcb-4e94-b73f-9a7911f0d8bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2nrm7\" (UID: \"63628291-cdcb-4e94-b73f-9a7911f0d8bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"
Oct 30 13:58:36.131256 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:36.131130    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45hwq\" (UniqueName: \"kubernetes.io/projected/63628291-cdcb-4e94-b73f-9a7911f0d8bb-kube-api-access-45hwq\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2nrm7\" (UID: \"63628291-cdcb-4e94-b73f-9a7911f0d8bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"
Oct 30 13:58:36.232726 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:36.232648    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63628291-cdcb-4e94-b73f-9a7911f0d8bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2nrm7\" (UID: \"63628291-cdcb-4e94-b73f-9a7911f0d8bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"
Oct 30 13:58:36.232726 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:36.232709    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45hwq\" (UniqueName: \"kubernetes.io/projected/63628291-cdcb-4e94-b73f-9a7911f0d8bb-kube-api-access-45hwq\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2nrm7\" (UID: \"63628291-cdcb-4e94-b73f-9a7911f0d8bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"
Oct 30 13:58:36.233085 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:36.233063    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63628291-cdcb-4e94-b73f-9a7911f0d8bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2nrm7\" (UID: \"63628291-cdcb-4e94-b73f-9a7911f0d8bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"
Oct 30 13:58:36.259461 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:36.259429    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45hwq\" (UniqueName: \"kubernetes.io/projected/63628291-cdcb-4e94-b73f-9a7911f0d8bb-kube-api-access-45hwq\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2nrm7\" (UID: \"63628291-cdcb-4e94-b73f-9a7911f0d8bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"
Oct 30 13:58:36.273185 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:36.273162    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"
Oct 30 13:58:36.599308 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:36.599281    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"]
Oct 30 13:58:36.610719 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:58:36.610668    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63628291_cdcb_4e94_b73f_9a7911f0d8bb.slice/crio-e4adcfe737112b66d7be1b4ba94c89b9f3aff7f39a50105bed397d62b3d01036 WatchSource:0}: Error finding container e4adcfe737112b66d7be1b4ba94c89b9f3aff7f39a50105bed397d62b3d01036: Status 404 returned error can't find the container with id e4adcfe737112b66d7be1b4ba94c89b9f3aff7f39a50105bed397d62b3d01036
Oct 30 13:58:37.503866 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:37.503810    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7" event={"ID":"63628291-cdcb-4e94-b73f-9a7911f0d8bb","Type":"ContainerStarted","Data":"e4adcfe737112b66d7be1b4ba94c89b9f3aff7f39a50105bed397d62b3d01036"}
Oct 30 13:58:38.581309 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:38.581277    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-rpn8z"]
Oct 30 13:58:38.591441 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:38.591417    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-rpn8z"
Oct 30 13:58:38.605145 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:38.605116    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-rpn8z"]
Oct 30 13:58:38.616608 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:38.616582    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-d4j9j\""
Oct 30 13:58:38.753665 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:38.753621    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fnq\" (UniqueName: \"kubernetes.io/projected/4b882a05-e44a-493d-86e6-171a423c8e2f-kube-api-access-22fnq\") pod \"authorino-operator-68d6dd76dc-rpn8z\" (UID: \"4b882a05-e44a-493d-86e6-171a423c8e2f\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-rpn8z"
Oct 30 13:58:38.854668 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:38.854578    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fnq\" (UniqueName: \"kubernetes.io/projected/4b882a05-e44a-493d-86e6-171a423c8e2f-kube-api-access-22fnq\") pod \"authorino-operator-68d6dd76dc-rpn8z\" (UID: \"4b882a05-e44a-493d-86e6-171a423c8e2f\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-rpn8z"
Oct 30 13:58:38.881329 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:38.881295    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fnq\" (UniqueName: \"kubernetes.io/projected/4b882a05-e44a-493d-86e6-171a423c8e2f-kube-api-access-22fnq\") pod \"authorino-operator-68d6dd76dc-rpn8z\" (UID: \"4b882a05-e44a-493d-86e6-171a423c8e2f\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-rpn8z"
Oct 30 13:58:38.910147 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:38.910119    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-rpn8z"
Oct 30 13:58:39.226168 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:39.226137    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-rpn8z"]
Oct 30 13:58:39.227771 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:58:39.227737    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b882a05_e44a_493d_86e6_171a423c8e2f.slice/crio-1d34467b4565e7eb4abccf6418f50af62dfc77ca88fd4c47003cc1d0d27ea408 WatchSource:0}: Error finding container 1d34467b4565e7eb4abccf6418f50af62dfc77ca88fd4c47003cc1d0d27ea408: Status 404 returned error can't find the container with id 1d34467b4565e7eb4abccf6418f50af62dfc77ca88fd4c47003cc1d0d27ea408
Oct 30 13:58:39.516999 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:39.516919    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-rpn8z" event={"ID":"4b882a05-e44a-493d-86e6-171a423c8e2f","Type":"ContainerStarted","Data":"1d34467b4565e7eb4abccf6418f50af62dfc77ca88fd4c47003cc1d0d27ea408"}
Oct 30 13:58:41.366120 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:58:41.366083    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:58:41.527838 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:41.527803    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7" event={"ID":"63628291-cdcb-4e94-b73f-9a7911f0d8bb","Type":"ContainerStarted","Data":"6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d"}
Oct 30 13:58:41.527988 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:41.527865    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"
Oct 30 13:58:41.607331 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:41.607287    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7" podStartSLOduration=2.638812253 podStartE2EDuration="6.607273161s" totalImagesPullingTime="3.968460908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:41.591591573 +0000 UTC m=+586.623490511" watchObservedRunningTime="2026-10-30 13:58:41.607273161 +0000 UTC m=+586.639172099"
Oct 30 13:58:43.538198 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:43.538163    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-rpn8z" event={"ID":"4b882a05-e44a-493d-86e6-171a423c8e2f","Type":"ContainerStarted","Data":"52ac5e4e55394704ce3d76cc6ce1fe8fe4d00ac20388e1e2f5023acc8bb2c14d"}
Oct 30 13:58:43.538576 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:43.538225    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-rpn8z"
Oct 30 13:58:43.586493 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:43.586447    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-rpn8z" podStartSLOduration=2.223149142 podStartE2EDuration="5.586433247s" totalImagesPullingTime="3.363284105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:43.584377283 +0000 UTC m=+588.616276221" watchObservedRunningTime="2026-10-30 13:58:43.586433247 +0000 UTC m=+588.618332185"
Oct 30 13:58:52.535191 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:52.535163    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"
Oct 30 13:58:54.545974 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:54.545945    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-rpn8z"
Oct 30 13:58:55.140380 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:55.140353    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 13:58:55.141301 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:55.141276    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 13:58:55.144827 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:55.144806    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 13:58:55.145749 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:55.145732    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 13:58:55.146145 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:55.146120    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 13:58:55.147050 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:58:55.147030    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 13:59:11.366050 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:59:11.366014    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:59:14.039418 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.039385    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"]
Oct 30 13:59:14.039951 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.039622    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7" podUID="63628291-cdcb-4e94-b73f-9a7911f0d8bb" containerName="manager" containerID="cri-o://6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d" gracePeriod=10
Oct 30 13:59:14.293277 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.293250    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"]
Oct 30 13:59:14.316626 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.316601    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"
Oct 30 13:59:14.322777 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.322736    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"]
Oct 30 13:59:14.322892 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.322840    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:14.433439 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.433415    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63628291-cdcb-4e94-b73f-9a7911f0d8bb-kube-api-access-45hwq\" (UniqueName: \"kubernetes.io/projected/63628291-cdcb-4e94-b73f-9a7911f0d8bb-kube-api-access-45hwq\") pod \"63628291-cdcb-4e94-b73f-9a7911f0d8bb\" (UID: \"63628291-cdcb-4e94-b73f-9a7911f0d8bb\") "
Oct 30 13:59:14.433643 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.433491    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63628291-cdcb-4e94-b73f-9a7911f0d8bb-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63628291-cdcb-4e94-b73f-9a7911f0d8bb-extensions-socket-volume\") pod \"63628291-cdcb-4e94-b73f-9a7911f0d8bb\" (UID: \"63628291-cdcb-4e94-b73f-9a7911f0d8bb\") "
Oct 30 13:59:14.433694 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.433670    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdg8w\" (UniqueName: \"kubernetes.io/projected/176bda91-3164-451b-af07-2bce1ed12489-kube-api-access-jdg8w\") pod \"kuadrant-operator-controller-manager-859cf9b48d-nkf6s\" (UID: \"176bda91-3164-451b-af07-2bce1ed12489\") " pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:14.433761 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.433742    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176bda91-3164-451b-af07-2bce1ed12489-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-859cf9b48d-nkf6s\" (UID: \"176bda91-3164-451b-af07-2bce1ed12489\") " pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:14.433857 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.433841    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63628291-cdcb-4e94-b73f-9a7911f0d8bb-extensions-socket-volume" pod "63628291-cdcb-4e94-b73f-9a7911f0d8bb" (UID: "63628291-cdcb-4e94-b73f-9a7911f0d8bb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:59:14.435580 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.435558    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63628291-cdcb-4e94-b73f-9a7911f0d8bb-kube-api-access-45hwq" pod "63628291-cdcb-4e94-b73f-9a7911f0d8bb" (UID: "63628291-cdcb-4e94-b73f-9a7911f0d8bb"). InnerVolumeSpecName "kube-api-access-45hwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:14.534083 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.534055    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176bda91-3164-451b-af07-2bce1ed12489-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-859cf9b48d-nkf6s\" (UID: \"176bda91-3164-451b-af07-2bce1ed12489\") " pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:14.534188 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.534114    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdg8w\" (UniqueName: \"kubernetes.io/projected/176bda91-3164-451b-af07-2bce1ed12489-kube-api-access-jdg8w\") pod \"kuadrant-operator-controller-manager-859cf9b48d-nkf6s\" (UID: \"176bda91-3164-451b-af07-2bce1ed12489\") " pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:14.534188 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.534151    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45hwq\" (UniqueName: \"kubernetes.io/projected/63628291-cdcb-4e94-b73f-9a7911f0d8bb-kube-api-access-45hwq\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:59:14.534188 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.534161    4530 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63628291-cdcb-4e94-b73f-9a7911f0d8bb-extensions-socket-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:59:14.534398 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.534381    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176bda91-3164-451b-af07-2bce1ed12489-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-859cf9b48d-nkf6s\" (UID: \"176bda91-3164-451b-af07-2bce1ed12489\") " pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:14.561660 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.561637    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdg8w\" (UniqueName: \"kubernetes.io/projected/176bda91-3164-451b-af07-2bce1ed12489-kube-api-access-jdg8w\") pod \"kuadrant-operator-controller-manager-859cf9b48d-nkf6s\" (UID: \"176bda91-3164-451b-af07-2bce1ed12489\") " pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:14.651879 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.651855    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:14.684617 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.684589    4530 generic.go:356] "Generic (PLEG): container finished" podID="63628291-cdcb-4e94-b73f-9a7911f0d8bb" containerID="6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d" exitCode=0
Oct 30 13:59:14.684695 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.684641    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"
Oct 30 13:59:14.684737 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.684684    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7" event={"ID":"63628291-cdcb-4e94-b73f-9a7911f0d8bb","Type":"ContainerDied","Data":"6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d"}
Oct 30 13:59:14.684779 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.684740    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7" event={"ID":"63628291-cdcb-4e94-b73f-9a7911f0d8bb","Type":"ContainerDied","Data":"e4adcfe737112b66d7be1b4ba94c89b9f3aff7f39a50105bed397d62b3d01036"}
Oct 30 13:59:14.684814 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.684777    4530 scope.go:122] "RemoveContainer" containerID="6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d"
Oct 30 13:59:14.694679 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.694651    4530 scope.go:122] "RemoveContainer" containerID="6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d"
Oct 30 13:59:14.695003 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:59:14.694981    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d\": container with ID starting with 6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d not found: ID does not exist" containerID="6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d"
Oct 30 13:59:14.695071 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.695012    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d"} err="failed to get container status \"6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d\": rpc error: code = NotFound desc = could not find container \"6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d\": container with ID starting with 6e968e86e663945f0d9c19b1ed8db54ed458459b4657cd725f6ee4353fe7a48d not found: ID does not exist"
Oct 30 13:59:14.746447 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.746380    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"]
Oct 30 13:59:14.752561 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.752516    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2nrm7"]
Oct 30 13:59:14.984628 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:14.984594    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"]
Oct 30 13:59:14.985494 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:59:14.985466    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176bda91_3164_451b_af07_2bce1ed12489.slice/crio-cc33e33a1ee6c843db2a59076e29b70b39295a2f0e0acbea956aaaa2dce00902 WatchSource:0}: Error finding container cc33e33a1ee6c843db2a59076e29b70b39295a2f0e0acbea956aaaa2dce00902: Status 404 returned error can't find the container with id cc33e33a1ee6c843db2a59076e29b70b39295a2f0e0acbea956aaaa2dce00902
Oct 30 13:59:15.113461 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:15.113433    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63628291-cdcb-4e94-b73f-9a7911f0d8bb" path="/var/lib/kubelet/pods/63628291-cdcb-4e94-b73f-9a7911f0d8bb/volumes"
Oct 30 13:59:15.691447 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:15.691404    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s" event={"ID":"176bda91-3164-451b-af07-2bce1ed12489","Type":"ContainerStarted","Data":"d278fd5e1a14876f42d98593003cf6f97f535096bda396d71d444c3dc322fb65"}
Oct 30 13:59:15.691640 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:15.691470    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:15.691640 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:15.691494    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s" event={"ID":"176bda91-3164-451b-af07-2bce1ed12489","Type":"ContainerStarted","Data":"cc33e33a1ee6c843db2a59076e29b70b39295a2f0e0acbea956aaaa2dce00902"}
Oct 30 13:59:15.770526 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:15.770482    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s" podStartSLOduration=1.7704689120000001 podStartE2EDuration="1.770468912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:59:15.769929339 +0000 UTC m=+620.801828279" watchObservedRunningTime="2026-10-30 13:59:15.770468912 +0000 UTC m=+620.802367851"
Oct 30 13:59:16.398078 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.398041    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"]
Oct 30 13:59:16.412408 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.412376    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"]
Oct 30 13:59:16.426352 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.426326    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"]
Oct 30 13:59:16.426684 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.426670    4530 cpu_manager.go:414] "containerMap: removing container" podUID="176bda91-3164-451b-af07-2bce1ed12489" containerName="manager"
Oct 30 13:59:16.426746 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.426686    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176bda91-3164-451b-af07-2bce1ed12489" containerName="manager"
Oct 30 13:59:16.426746 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.426711    4530 cpu_manager.go:414] "containerMap: removing container" podUID="63628291-cdcb-4e94-b73f-9a7911f0d8bb" containerName="manager"
Oct 30 13:59:16.426746 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.426717    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63628291-cdcb-4e94-b73f-9a7911f0d8bb" containerName="manager"
Oct 30 13:59:16.426840 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.426775    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="176bda91-3164-451b-af07-2bce1ed12489" containerName="manager"
Oct 30 13:59:16.426840 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.426789    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="63628291-cdcb-4e94-b73f-9a7911f0d8bb" containerName="manager"
Oct 30 13:59:16.428773 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.428753    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"
Oct 30 13:59:16.455286 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.455260    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"]
Oct 30 13:59:16.473388 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.473365    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"]
Oct 30 13:59:16.476216 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.476193    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"
Oct 30 13:59:16.502554 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.502517    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"]
Oct 30 13:59:16.550844 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.550815    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74jtq\" (UniqueName: \"kubernetes.io/projected/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-kube-api-access-74jtq\") pod \"kuadrant-operator-controller-manager-956956bc4-gvs4f\" (UID: \"39f36e4a-8fee-41bb-8aa4-1438b9cadf13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"
Oct 30 13:59:16.550981 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.550882    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-859cf9b48d-lhj7b\" (UID: \"6c8337d8-7cb9-4b2b-81a3-4c58e092b341\") " pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"
Oct 30 13:59:16.550981 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.550928    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-gvs4f\" (UID: \"39f36e4a-8fee-41bb-8aa4-1438b9cadf13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"
Oct 30 13:59:16.551071 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.550997    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn24x\" (UniqueName: \"kubernetes.io/projected/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-kube-api-access-kn24x\") pod \"kuadrant-operator-controller-manager-859cf9b48d-lhj7b\" (UID: \"6c8337d8-7cb9-4b2b-81a3-4c58e092b341\") " pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"
Oct 30 13:59:16.651821 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.651752    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-gvs4f\" (UID: \"39f36e4a-8fee-41bb-8aa4-1438b9cadf13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"
Oct 30 13:59:16.651821 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.651784    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn24x\" (UniqueName: \"kubernetes.io/projected/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-kube-api-access-kn24x\") pod \"kuadrant-operator-controller-manager-859cf9b48d-lhj7b\" (UID: \"6c8337d8-7cb9-4b2b-81a3-4c58e092b341\") " pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"
Oct 30 13:59:16.651989 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.651825    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74jtq\" (UniqueName: \"kubernetes.io/projected/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-kube-api-access-74jtq\") pod \"kuadrant-operator-controller-manager-956956bc4-gvs4f\" (UID: \"39f36e4a-8fee-41bb-8aa4-1438b9cadf13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"
Oct 30 13:59:16.651989 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.651865    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-859cf9b48d-lhj7b\" (UID: \"6c8337d8-7cb9-4b2b-81a3-4c58e092b341\") " pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"
Oct 30 13:59:16.652115 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.652090    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-gvs4f\" (UID: \"39f36e4a-8fee-41bb-8aa4-1438b9cadf13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"
Oct 30 13:59:16.652243 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.652219    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-859cf9b48d-lhj7b\" (UID: \"6c8337d8-7cb9-4b2b-81a3-4c58e092b341\") " pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"
Oct 30 13:59:16.678632 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.678607    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74jtq\" (UniqueName: \"kubernetes.io/projected/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-kube-api-access-74jtq\") pod \"kuadrant-operator-controller-manager-956956bc4-gvs4f\" (UID: \"39f36e4a-8fee-41bb-8aa4-1438b9cadf13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"
Oct 30 13:59:16.678755 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.678677    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn24x\" (UniqueName: \"kubernetes.io/projected/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-kube-api-access-kn24x\") pod \"kuadrant-operator-controller-manager-859cf9b48d-lhj7b\" (UID: \"6c8337d8-7cb9-4b2b-81a3-4c58e092b341\") " pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"
Oct 30 13:59:16.741209 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.741185    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"
Oct 30 13:59:16.790662 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:16.790628    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"
Oct 30 13:59:17.065925 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:59:17.065896    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8337d8_7cb9_4b2b_81a3_4c58e092b341.slice/crio-a5802250187272eefe0bb81163764111de1588e7d6ff3def005f167edb2cb4a8 WatchSource:0}: Error finding container a5802250187272eefe0bb81163764111de1588e7d6ff3def005f167edb2cb4a8: Status 404 returned error can't find the container with id a5802250187272eefe0bb81163764111de1588e7d6ff3def005f167edb2cb4a8
Oct 30 13:59:17.067574 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:17.067522    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"]
Oct 30 13:59:17.117843 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:59:17.117802    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f36e4a_8fee_41bb_8aa4_1438b9cadf13.slice/crio-4bdb81f6330e17b743fc030970a2e077ce81cd4579282d4ce3141c80285ccfce WatchSource:0}: Error finding container 4bdb81f6330e17b743fc030970a2e077ce81cd4579282d4ce3141c80285ccfce: Status 404 returned error can't find the container with id 4bdb81f6330e17b743fc030970a2e077ce81cd4579282d4ce3141c80285ccfce
Oct 30 13:59:17.118529 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:17.118507    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"]
Oct 30 13:59:17.704577 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:17.704512    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b" event={"ID":"6c8337d8-7cb9-4b2b-81a3-4c58e092b341","Type":"ContainerStarted","Data":"b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58"}
Oct 30 13:59:17.704999 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:17.704600    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b" event={"ID":"6c8337d8-7cb9-4b2b-81a3-4c58e092b341","Type":"ContainerStarted","Data":"a5802250187272eefe0bb81163764111de1588e7d6ff3def005f167edb2cb4a8"}
Oct 30 13:59:17.704999 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:17.704626    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"
Oct 30 13:59:17.706310 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:17.706278    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f" event={"ID":"39f36e4a-8fee-41bb-8aa4-1438b9cadf13","Type":"ContainerStarted","Data":"999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153"}
Oct 30 13:59:17.706418 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:17.706325    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f" event={"ID":"39f36e4a-8fee-41bb-8aa4-1438b9cadf13","Type":"ContainerStarted","Data":"4bdb81f6330e17b743fc030970a2e077ce81cd4579282d4ce3141c80285ccfce"}
Oct 30 13:59:17.706418 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:17.706337    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s" podUID="176bda91-3164-451b-af07-2bce1ed12489" containerName="manager" containerID="cri-o://d278fd5e1a14876f42d98593003cf6f97f535096bda396d71d444c3dc322fb65" gracePeriod=2
Oct 30 13:59:17.706532 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:17.706469    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"
Oct 30 13:59:17.763852 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:17.763813    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b" podStartSLOduration=1.763800948 podStartE2EDuration="1.763800948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:59:17.761795227 +0000 UTC m=+622.793694165" watchObservedRunningTime="2026-10-30 13:59:17.763800948 +0000 UTC m=+622.795699887"
Oct 30 13:59:17.812212 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:17.812171    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f" podStartSLOduration=1.812158828 podStartE2EDuration="1.812158828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:59:17.810831781 +0000 UTC m=+622.842730721" watchObservedRunningTime="2026-10-30 13:59:17.812158828 +0000 UTC m=+622.844057766"
Oct 30 13:59:17.951109 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:17.951080    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:18.062958 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:18.062932    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176bda91-3164-451b-af07-2bce1ed12489-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176bda91-3164-451b-af07-2bce1ed12489-extensions-socket-volume\") pod \"176bda91-3164-451b-af07-2bce1ed12489\" (UID: \"176bda91-3164-451b-af07-2bce1ed12489\") "
Oct 30 13:59:18.063082 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:18.062994    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176bda91-3164-451b-af07-2bce1ed12489-kube-api-access-jdg8w\" (UniqueName: \"kubernetes.io/projected/176bda91-3164-451b-af07-2bce1ed12489-kube-api-access-jdg8w\") pod \"176bda91-3164-451b-af07-2bce1ed12489\" (UID: \"176bda91-3164-451b-af07-2bce1ed12489\") "
Oct 30 13:59:18.063301 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:18.063274    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176bda91-3164-451b-af07-2bce1ed12489-extensions-socket-volume" pod "176bda91-3164-451b-af07-2bce1ed12489" (UID: "176bda91-3164-451b-af07-2bce1ed12489"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:59:18.065018 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:18.064998    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176bda91-3164-451b-af07-2bce1ed12489-kube-api-access-jdg8w" pod "176bda91-3164-451b-af07-2bce1ed12489" (UID: "176bda91-3164-451b-af07-2bce1ed12489"). InnerVolumeSpecName "kube-api-access-jdg8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:18.163691 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:18.163663    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdg8w\" (UniqueName: \"kubernetes.io/projected/176bda91-3164-451b-af07-2bce1ed12489-kube-api-access-jdg8w\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:59:18.163691 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:18.163687    4530 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176bda91-3164-451b-af07-2bce1ed12489-extensions-socket-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:59:18.712783 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:18.712743    4530 generic.go:356] "Generic (PLEG): container finished" podID="176bda91-3164-451b-af07-2bce1ed12489" containerID="d278fd5e1a14876f42d98593003cf6f97f535096bda396d71d444c3dc322fb65" exitCode=0
Oct 30 13:59:18.713180 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:18.712799    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:18.713180 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:18.712838    4530 scope.go:122] "RemoveContainer" containerID="d278fd5e1a14876f42d98593003cf6f97f535096bda396d71d444c3dc322fb65"
Oct 30 13:59:18.721641 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:18.721624    4530 scope.go:122] "RemoveContainer" containerID="d278fd5e1a14876f42d98593003cf6f97f535096bda396d71d444c3dc322fb65"
Oct 30 13:59:18.721866 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:59:18.721851    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d278fd5e1a14876f42d98593003cf6f97f535096bda396d71d444c3dc322fb65\": container with ID starting with d278fd5e1a14876f42d98593003cf6f97f535096bda396d71d444c3dc322fb65 not found: ID does not exist" containerID="d278fd5e1a14876f42d98593003cf6f97f535096bda396d71d444c3dc322fb65"
Oct 30 13:59:18.721905 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:18.721873    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d278fd5e1a14876f42d98593003cf6f97f535096bda396d71d444c3dc322fb65"} err="failed to get container status \"d278fd5e1a14876f42d98593003cf6f97f535096bda396d71d444c3dc322fb65\": rpc error: code = NotFound desc = could not find container \"d278fd5e1a14876f42d98593003cf6f97f535096bda396d71d444c3dc322fb65\": container with ID starting with d278fd5e1a14876f42d98593003cf6f97f535096bda396d71d444c3dc322fb65 not found: ID does not exist"
Oct 30 13:59:18.728221 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:59:18.728195    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-859cf9b48d-nkf6s\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="176bda91-3164-451b-af07-2bce1ed12489" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:18.743583 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:59:18.743559    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-859cf9b48d-nkf6s\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="176bda91-3164-451b-af07-2bce1ed12489" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-nkf6s"
Oct 30 13:59:19.112373 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:19.112341    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176bda91-3164-451b-af07-2bce1ed12489" path="/var/lib/kubelet/pods/176bda91-3164-451b-af07-2bce1ed12489/volumes"
Oct 30 13:59:28.715685 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:28.715655    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"
Oct 30 13:59:28.716196 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:28.715804    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"
Oct 30 13:59:28.827899 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:28.827867    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"]
Oct 30 13:59:28.828055 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:28.828030    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f" podUID="39f36e4a-8fee-41bb-8aa4-1438b9cadf13" containerName="manager" containerID="cri-o://999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153" gracePeriod=10
Oct 30 13:59:29.075508 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.075482    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"
Oct 30 13:59:29.136137 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.136109    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-kube-api-access-74jtq\" (UniqueName: \"kubernetes.io/projected/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-kube-api-access-74jtq\") pod \"39f36e4a-8fee-41bb-8aa4-1438b9cadf13\" (UID: \"39f36e4a-8fee-41bb-8aa4-1438b9cadf13\") "
Oct 30 13:59:29.136296 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.136153    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-extensions-socket-volume\") pod \"39f36e4a-8fee-41bb-8aa4-1438b9cadf13\" (UID: \"39f36e4a-8fee-41bb-8aa4-1438b9cadf13\") "
Oct 30 13:59:29.136530 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.136507    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-extensions-socket-volume" pod "39f36e4a-8fee-41bb-8aa4-1438b9cadf13" (UID: "39f36e4a-8fee-41bb-8aa4-1438b9cadf13"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:59:29.138220 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.138197    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-kube-api-access-74jtq" pod "39f36e4a-8fee-41bb-8aa4-1438b9cadf13" (UID: "39f36e4a-8fee-41bb-8aa4-1438b9cadf13"). InnerVolumeSpecName "kube-api-access-74jtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:29.237172 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.237137    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74jtq\" (UniqueName: \"kubernetes.io/projected/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-kube-api-access-74jtq\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:59:29.237172 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.237163    4530 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39f36e4a-8fee-41bb-8aa4-1438b9cadf13-extensions-socket-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:59:29.768278 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.768246    4530 generic.go:356] "Generic (PLEG): container finished" podID="39f36e4a-8fee-41bb-8aa4-1438b9cadf13" containerID="999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153" exitCode=0
Oct 30 13:59:29.768748 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.768293    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"
Oct 30 13:59:29.768748 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.768332    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f" event={"ID":"39f36e4a-8fee-41bb-8aa4-1438b9cadf13","Type":"ContainerDied","Data":"999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153"}
Oct 30 13:59:29.768748 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.768396    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f" event={"ID":"39f36e4a-8fee-41bb-8aa4-1438b9cadf13","Type":"ContainerDied","Data":"4bdb81f6330e17b743fc030970a2e077ce81cd4579282d4ce3141c80285ccfce"}
Oct 30 13:59:29.768748 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.768430    4530 scope.go:122] "RemoveContainer" containerID="999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153"
Oct 30 13:59:29.777042 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.777026    4530 scope.go:122] "RemoveContainer" containerID="999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153"
Oct 30 13:59:29.777281 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:59:29.777262    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153\": container with ID starting with 999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153 not found: ID does not exist" containerID="999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153"
Oct 30 13:59:29.777321 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.777291    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153"} err="failed to get container status \"999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153\": rpc error: code = NotFound desc = could not find container \"999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153\": container with ID starting with 999df19de3dd60d31c5364fc6674109efa80b75028036d10c44646518f978153 not found: ID does not exist"
Oct 30 13:59:29.858044 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.858019    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"]
Oct 30 13:59:29.863965 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:29.863944    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-gvs4f"]
Oct 30 13:59:31.112428 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:31.112398    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f36e4a-8fee-41bb-8aa4-1438b9cadf13" path="/var/lib/kubelet/pods/39f36e4a-8fee-41bb-8aa4-1438b9cadf13/volumes"
Oct 30 13:59:41.146092 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.146059    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"]
Oct 30 13:59:41.146501 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.146264    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b" podUID="6c8337d8-7cb9-4b2b-81a3-4c58e092b341" containerName="manager" containerID="cri-o://b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58" gracePeriod=10
Oct 30 13:59:41.365500 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:59:41.365467    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 13:59:41.387837 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.387812    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"
Oct 30 13:59:41.523786 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.523727    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-kube-api-access-kn24x\" (UniqueName: \"kubernetes.io/projected/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-kube-api-access-kn24x\") pod \"6c8337d8-7cb9-4b2b-81a3-4c58e092b341\" (UID: \"6c8337d8-7cb9-4b2b-81a3-4c58e092b341\") "
Oct 30 13:59:41.523903 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.523791    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-extensions-socket-volume\") pod \"6c8337d8-7cb9-4b2b-81a3-4c58e092b341\" (UID: \"6c8337d8-7cb9-4b2b-81a3-4c58e092b341\") "
Oct 30 13:59:41.524154 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.524132    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-extensions-socket-volume" pod "6c8337d8-7cb9-4b2b-81a3-4c58e092b341" (UID: "6c8337d8-7cb9-4b2b-81a3-4c58e092b341"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:59:41.525692 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.525668    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-kube-api-access-kn24x" pod "6c8337d8-7cb9-4b2b-81a3-4c58e092b341" (UID: "6c8337d8-7cb9-4b2b-81a3-4c58e092b341"). InnerVolumeSpecName "kube-api-access-kn24x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:41.624363 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.624341    4530 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-extensions-socket-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:59:41.624363 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.624362    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn24x\" (UniqueName: \"kubernetes.io/projected/6c8337d8-7cb9-4b2b-81a3-4c58e092b341-kube-api-access-kn24x\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 13:59:41.834696 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.834670    4530 generic.go:356] "Generic (PLEG): container finished" podID="6c8337d8-7cb9-4b2b-81a3-4c58e092b341" containerID="b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58" exitCode=0
Oct 30 13:59:41.834821 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.834714    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"
Oct 30 13:59:41.834821 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.834732    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b" event={"ID":"6c8337d8-7cb9-4b2b-81a3-4c58e092b341","Type":"ContainerDied","Data":"b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58"}
Oct 30 13:59:41.834821 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.834771    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b" event={"ID":"6c8337d8-7cb9-4b2b-81a3-4c58e092b341","Type":"ContainerDied","Data":"a5802250187272eefe0bb81163764111de1588e7d6ff3def005f167edb2cb4a8"}
Oct 30 13:59:41.834821 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.834793    4530 scope.go:122] "RemoveContainer" containerID="b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58"
Oct 30 13:59:41.843513 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.843497    4530 scope.go:122] "RemoveContainer" containerID="b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58"
Oct 30 13:59:41.843803 ip-10-0-133-118 kubenswrapper[4530]: E1030 13:59:41.843782    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58\": container with ID starting with b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58 not found: ID does not exist" containerID="b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58"
Oct 30 13:59:41.843857 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.843812    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58"} err="failed to get container status \"b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58\": rpc error: code = NotFound desc = could not find container \"b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58\": container with ID starting with b0f8f0312bdffd48f4e3fa43e8dacaba6e3007fa134881d6b0af2d50562c4b58 not found: ID does not exist"
Oct 30 13:59:41.902607 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.902583    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"]
Oct 30 13:59:41.908078 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:41.908054    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-859cf9b48d-lhj7b"]
Oct 30 13:59:43.112655 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:43.112624    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8337d8-7cb9-4b2b-81a3-4c58e092b341" path="/var/lib/kubelet/pods/6c8337d8-7cb9-4b2b-81a3-4c58e092b341/volumes"
Oct 30 13:59:46.129187 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.129155    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"]
Oct 30 13:59:46.129676 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.129460    4530 cpu_manager.go:414] "containerMap: removing container" podUID="39f36e4a-8fee-41bb-8aa4-1438b9cadf13" containerName="manager"
Oct 30 13:59:46.129676 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.129472    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f36e4a-8fee-41bb-8aa4-1438b9cadf13" containerName="manager"
Oct 30 13:59:46.129676 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.129492    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6c8337d8-7cb9-4b2b-81a3-4c58e092b341" containerName="manager"
Oct 30 13:59:46.129676 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.129496    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8337d8-7cb9-4b2b-81a3-4c58e092b341" containerName="manager"
Oct 30 13:59:46.129676 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.129566    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f36e4a-8fee-41bb-8aa4-1438b9cadf13" containerName="manager"
Oct 30 13:59:46.129676 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.129578    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8337d8-7cb9-4b2b-81a3-4c58e092b341" containerName="manager"
Oct 30 13:59:46.132603 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.132583    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.155293 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.155265    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"]
Oct 30 13:59:46.166426 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.166402    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5drtm\""
Oct 30 13:59:46.261473 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.261442    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.261635 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.261599    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39d4963-4125-44ec-aa19-020b8e287167-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.261696 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.261652    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39d4963-4125-44ec-aa19-020b8e287167-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.261696 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.261673    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.261804 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.261790    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.261857 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.261825    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.261911 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.261852    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqx5z\" (UniqueName: \"kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-kube-api-access-tqx5z\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.261911 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.261899    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.262006 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.261921    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.362390 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.362362    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.362498 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.362405    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39d4963-4125-44ec-aa19-020b8e287167-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.362572 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.362522    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39d4963-4125-44ec-aa19-020b8e287167-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.362636 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.362577    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.362691 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.362673    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.362759 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.362715    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.362759 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.362744    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqx5z\" (UniqueName: \"kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-kube-api-access-tqx5z\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.362865 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.362773    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.362865 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.362801    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.362970 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.362892    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.363181 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.363155    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.363263 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.363186    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.363263 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.363202    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.363361 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.363303    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39d4963-4125-44ec-aa19-020b8e287167-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.364722 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.364701    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.364883 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.364866    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39d4963-4125-44ec-aa19-020b8e287167-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.389266 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.389215    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.389739 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.389722    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqx5z\" (UniqueName: \"kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-kube-api-access-tqx5z\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-94bz7\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.445771 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.445750    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:46.757743 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.757720    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"]
Oct 30 13:59:46.758592 ip-10-0-133-118 kubenswrapper[4530]: W1030 13:59:46.758565    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39d4963_4125_44ec_aa19_020b8e287167.slice/crio-ddb3ff09e06c1427c0644fa1a05e526f82e3f276ed5e3e45032e50bcf9815e6b WatchSource:0}: Error finding container ddb3ff09e06c1427c0644fa1a05e526f82e3f276ed5e3e45032e50bcf9815e6b: Status 404 returned error can't find the container with id ddb3ff09e06c1427c0644fa1a05e526f82e3f276ed5e3e45032e50bcf9815e6b
Oct 30 13:59:46.760370 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.760347    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:59:46.865254 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:46.865226    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" event={"ID":"b39d4963-4125-44ec-aa19-020b8e287167","Type":"ContainerStarted","Data":"ddb3ff09e06c1427c0644fa1a05e526f82e3f276ed5e3e45032e50bcf9815e6b"}
Oct 30 13:59:49.342419 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:49.342388    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:59:49.342678 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:49.342454    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:59:49.342678 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:49.342482    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:59:49.883366 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:49.883326    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" event={"ID":"b39d4963-4125-44ec-aa19-020b8e287167","Type":"ContainerStarted","Data":"fb0496343822c7dabae06ca9080c89d176c15a697b5459512fea8abd8fe8ea89"}
Oct 30 13:59:49.948948 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:49.948904    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" podStartSLOduration=1.367783223 podStartE2EDuration="3.948892154s" totalImagesPullingTime="2.581108931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:59:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:59:49.948449752 +0000 UTC m=+654.980348692" watchObservedRunningTime="2026-10-30 13:59:49.948892154 +0000 UTC m=+654.980791093"
Oct 30 13:59:50.446475 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:50.446446    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:50.451171 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:50.451147    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:50.888645 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:50.888614    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 13:59:50.889487 ip-10-0-133-118 kubenswrapper[4530]: I1030 13:59:50.889466    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 14:00:01.915730 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:01.915692    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"]
Oct 30 14:00:01.919622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:01.919604    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"
Oct 30 14:00:01.937367 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:01.937342    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"]
Oct 30 14:00:01.949301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:01.949277    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 14:00:01.949420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:01.949393    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-x9mcf\""
Oct 30 14:00:02.048633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:02.048599    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"]
Oct 30 14:00:02.079652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:02.079623    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/183cc0e1-7303-4b9a-b9cb-8753c38db696-config-file\") pod \"limitador-limitador-74dfcfb894-mvt8w\" (UID: \"183cc0e1-7303-4b9a-b9cb-8753c38db696\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"
Oct 30 14:00:02.079779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:02.079677    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgz8p\" (UniqueName: \"kubernetes.io/projected/183cc0e1-7303-4b9a-b9cb-8753c38db696-kube-api-access-cgz8p\") pod \"limitador-limitador-74dfcfb894-mvt8w\" (UID: \"183cc0e1-7303-4b9a-b9cb-8753c38db696\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"
Oct 30 14:00:02.181032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:02.180956    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/183cc0e1-7303-4b9a-b9cb-8753c38db696-config-file\") pod \"limitador-limitador-74dfcfb894-mvt8w\" (UID: \"183cc0e1-7303-4b9a-b9cb-8753c38db696\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"
Oct 30 14:00:02.181032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:02.181017    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgz8p\" (UniqueName: \"kubernetes.io/projected/183cc0e1-7303-4b9a-b9cb-8753c38db696-kube-api-access-cgz8p\") pod \"limitador-limitador-74dfcfb894-mvt8w\" (UID: \"183cc0e1-7303-4b9a-b9cb-8753c38db696\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"
Oct 30 14:00:02.181616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:02.181595    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/183cc0e1-7303-4b9a-b9cb-8753c38db696-config-file\") pod \"limitador-limitador-74dfcfb894-mvt8w\" (UID: \"183cc0e1-7303-4b9a-b9cb-8753c38db696\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"
Oct 30 14:00:02.213577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:02.213529    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgz8p\" (UniqueName: \"kubernetes.io/projected/183cc0e1-7303-4b9a-b9cb-8753c38db696-kube-api-access-cgz8p\") pod \"limitador-limitador-74dfcfb894-mvt8w\" (UID: \"183cc0e1-7303-4b9a-b9cb-8753c38db696\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"
Oct 30 14:00:02.231242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:02.231212    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"
Oct 30 14:00:02.529111 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:00:02.529077    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183cc0e1_7303_4b9a_b9cb_8753c38db696.slice/crio-942a5d310d5e570aeaeb9bfc25661c84b02adcfb6dbb7599f96f070af2d3e6f6 WatchSource:0}: Error finding container 942a5d310d5e570aeaeb9bfc25661c84b02adcfb6dbb7599f96f070af2d3e6f6: Status 404 returned error can't find the container with id 942a5d310d5e570aeaeb9bfc25661c84b02adcfb6dbb7599f96f070af2d3e6f6
Oct 30 14:00:02.529310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:02.529288    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"]
Oct 30 14:00:02.947286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:02.947255    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w" event={"ID":"183cc0e1-7303-4b9a-b9cb-8753c38db696","Type":"ContainerStarted","Data":"942a5d310d5e570aeaeb9bfc25661c84b02adcfb6dbb7599f96f070af2d3e6f6"}
Oct 30 14:00:03.471699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:03.471658    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:00:03.474785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:03.474761    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"
Oct 30 14:00:03.496813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:03.496789    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:00:03.570266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:03.570241    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:00:03.593120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:03.593096    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f95m4\" (UniqueName: \"kubernetes.io/projected/3ddceb78-7035-4583-a501-3552ad3bc858-kube-api-access-f95m4\") pod \"limitador-limitador-7c9b87d574-lr8xc\" (UID: \"3ddceb78-7035-4583-a501-3552ad3bc858\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"
Oct 30 14:00:03.593236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:03.593217    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ddceb78-7035-4583-a501-3552ad3bc858-config-file\") pod \"limitador-limitador-7c9b87d574-lr8xc\" (UID: \"3ddceb78-7035-4583-a501-3552ad3bc858\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"
Oct 30 14:00:03.693765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:03.693737    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ddceb78-7035-4583-a501-3552ad3bc858-config-file\") pod \"limitador-limitador-7c9b87d574-lr8xc\" (UID: \"3ddceb78-7035-4583-a501-3552ad3bc858\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"
Oct 30 14:00:03.693880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:03.693771    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f95m4\" (UniqueName: \"kubernetes.io/projected/3ddceb78-7035-4583-a501-3552ad3bc858-kube-api-access-f95m4\") pod \"limitador-limitador-7c9b87d574-lr8xc\" (UID: \"3ddceb78-7035-4583-a501-3552ad3bc858\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"
Oct 30 14:00:03.694322 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:03.694304    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ddceb78-7035-4583-a501-3552ad3bc858-config-file\") pod \"limitador-limitador-7c9b87d574-lr8xc\" (UID: \"3ddceb78-7035-4583-a501-3552ad3bc858\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"
Oct 30 14:00:03.720681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:03.720658    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f95m4\" (UniqueName: \"kubernetes.io/projected/3ddceb78-7035-4583-a501-3552ad3bc858-kube-api-access-f95m4\") pod \"limitador-limitador-7c9b87d574-lr8xc\" (UID: \"3ddceb78-7035-4583-a501-3552ad3bc858\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"
Oct 30 14:00:03.787255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:03.787197    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"
Oct 30 14:00:04.086105 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:00:04.086076    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ddceb78_7035_4583_a501_3552ad3bc858.slice/crio-69d13770fa2eb91814d1ebdd18589eaa32cd2069b35880ad251fd8ce8a71cff7 WatchSource:0}: Error finding container 69d13770fa2eb91814d1ebdd18589eaa32cd2069b35880ad251fd8ce8a71cff7: Status 404 returned error can't find the container with id 69d13770fa2eb91814d1ebdd18589eaa32cd2069b35880ad251fd8ce8a71cff7
Oct 30 14:00:04.086509 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:04.086159    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:00:04.960968 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:04.960933    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc" event={"ID":"3ddceb78-7035-4583-a501-3552ad3bc858","Type":"ContainerStarted","Data":"69d13770fa2eb91814d1ebdd18589eaa32cd2069b35880ad251fd8ce8a71cff7"}
Oct 30 14:00:05.967309 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:05.967271    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w" event={"ID":"183cc0e1-7303-4b9a-b9cb-8753c38db696","Type":"ContainerStarted","Data":"3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b"}
Oct 30 14:00:05.967741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:05.967414    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"
Oct 30 14:00:06.015452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:06.015406    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w" podStartSLOduration=2.304793267 podStartE2EDuration="5.015395426s" totalImagesPullingTime="2.710602159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:06.013776903 +0000 UTC m=+671.045675843" watchObservedRunningTime="2026-10-30 14:00:06.015395426 +0000 UTC m=+671.047294364"
Oct 30 14:00:06.974408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:06.974324    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc" event={"ID":"3ddceb78-7035-4583-a501-3552ad3bc858","Type":"ContainerStarted","Data":"2f492575090aa523f559d213ab12fe62444ea2bba2777d288e4ea11fc2c3c5f5"}
Oct 30 14:00:06.974819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:06.974568    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"
Oct 30 14:00:07.021934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:07.021893    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc" podStartSLOduration=1.386994841 podStartE2EDuration="4.021881696s" totalImagesPullingTime="2.634886855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:07.020981148 +0000 UTC m=+672.052880087" watchObservedRunningTime="2026-10-30 14:00:07.021881696 +0000 UTC m=+672.053780635"
Oct 30 14:00:11.365468 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:00:11.365393    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:00:16.975276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:16.975244    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"
Oct 30 14:00:17.979634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:17.979603    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"
Oct 30 14:00:18.065843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:18.065800    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"]
Oct 30 14:00:18.066028 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:18.065995    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w" podUID="183cc0e1-7303-4b9a-b9cb-8753c38db696" containerName="limitador" containerID="cri-o://3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b" gracePeriod=30
Oct 30 14:00:18.611143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:18.611119    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"
Oct 30 14:00:18.709527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:18.709462    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/183cc0e1-7303-4b9a-b9cb-8753c38db696-config-file\" (UniqueName: \"kubernetes.io/configmap/183cc0e1-7303-4b9a-b9cb-8753c38db696-config-file\") pod \"183cc0e1-7303-4b9a-b9cb-8753c38db696\" (UID: \"183cc0e1-7303-4b9a-b9cb-8753c38db696\") "
Oct 30 14:00:18.709810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:18.709787    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183cc0e1-7303-4b9a-b9cb-8753c38db696-config-file" pod "183cc0e1-7303-4b9a-b9cb-8753c38db696" (UID: "183cc0e1-7303-4b9a-b9cb-8753c38db696"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:18.809930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:18.809904    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183cc0e1-7303-4b9a-b9cb-8753c38db696-kube-api-access-cgz8p\" (UniqueName: \"kubernetes.io/projected/183cc0e1-7303-4b9a-b9cb-8753c38db696-kube-api-access-cgz8p\") pod \"183cc0e1-7303-4b9a-b9cb-8753c38db696\" (UID: \"183cc0e1-7303-4b9a-b9cb-8753c38db696\") "
Oct 30 14:00:18.810134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:18.810117    4530 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/183cc0e1-7303-4b9a-b9cb-8753c38db696-config-file\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:00:18.811929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:18.811907    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183cc0e1-7303-4b9a-b9cb-8753c38db696-kube-api-access-cgz8p" pod "183cc0e1-7303-4b9a-b9cb-8753c38db696" (UID: "183cc0e1-7303-4b9a-b9cb-8753c38db696"). InnerVolumeSpecName "kube-api-access-cgz8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:18.911228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:18.911202    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgz8p\" (UniqueName: \"kubernetes.io/projected/183cc0e1-7303-4b9a-b9cb-8753c38db696-kube-api-access-cgz8p\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:00:19.034948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:19.034875    4530 generic.go:356] "Generic (PLEG): container finished" podID="183cc0e1-7303-4b9a-b9cb-8753c38db696" containerID="3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b" exitCode=0
Oct 30 14:00:19.034948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:19.034918    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"
Oct 30 14:00:19.035372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:19.034955    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w" event={"ID":"183cc0e1-7303-4b9a-b9cb-8753c38db696","Type":"ContainerDied","Data":"3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b"}
Oct 30 14:00:19.035372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:19.035008    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-mvt8w" event={"ID":"183cc0e1-7303-4b9a-b9cb-8753c38db696","Type":"ContainerDied","Data":"942a5d310d5e570aeaeb9bfc25661c84b02adcfb6dbb7599f96f070af2d3e6f6"}
Oct 30 14:00:19.035372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:19.035030    4530 scope.go:122] "RemoveContainer" containerID="3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b"
Oct 30 14:00:19.043523 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:19.043509    4530 scope.go:122] "RemoveContainer" containerID="3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b"
Oct 30 14:00:19.043806 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:00:19.043788    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b\": container with ID starting with 3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b not found: ID does not exist" containerID="3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b"
Oct 30 14:00:19.043886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:19.043814    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b"} err="failed to get container status \"3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b\": rpc error: code = NotFound desc = could not find container \"3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b\": container with ID starting with 3d54eaa1a23f72e0cb72c63d9f5382cba262b254f453572cd8a791706e87026b not found: ID does not exist"
Oct 30 14:00:19.095787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:19.095761    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"]
Oct 30 14:00:19.112425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:19.112404    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-mvt8w"]
Oct 30 14:00:21.112273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:21.112244    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="183cc0e1-7303-4b9a-b9cb-8753c38db696" path="/var/lib/kubelet/pods/183cc0e1-7303-4b9a-b9cb-8753c38db696/volumes"
Oct 30 14:00:34.320990 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.320956    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"]
Oct 30 14:00:34.321460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.321267    4530 cpu_manager.go:414] "containerMap: removing container" podUID="183cc0e1-7303-4b9a-b9cb-8753c38db696" containerName="limitador"
Oct 30 14:00:34.321460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.321278    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183cc0e1-7303-4b9a-b9cb-8753c38db696" containerName="limitador"
Oct 30 14:00:34.321460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.321366    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="183cc0e1-7303-4b9a-b9cb-8753c38db696" containerName="limitador"
Oct 30 14:00:34.325935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.325919    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"
Oct 30 14:00:34.352654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.352626    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"]
Oct 30 14:00:34.362701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.362676    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:00:34.362814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.362747    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:00:34.363025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.363007    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9mcbf\""
Oct 30 14:00:34.363073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.363032    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 14:00:34.429953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.429926    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f937b72b-9f30-4465-bb35-f17afd3c5335-data\") pod \"postgres-7cdbf6dc44-62qgf\" (UID: \"f937b72b-9f30-4465-bb35-f17afd3c5335\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"
Oct 30 14:00:34.430061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.429970    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4dsb\" (UniqueName: \"kubernetes.io/projected/f937b72b-9f30-4465-bb35-f17afd3c5335-kube-api-access-h4dsb\") pod \"postgres-7cdbf6dc44-62qgf\" (UID: \"f937b72b-9f30-4465-bb35-f17afd3c5335\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"
Oct 30 14:00:34.530409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.530381    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f937b72b-9f30-4465-bb35-f17afd3c5335-data\") pod \"postgres-7cdbf6dc44-62qgf\" (UID: \"f937b72b-9f30-4465-bb35-f17afd3c5335\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"
Oct 30 14:00:34.530521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.530421    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4dsb\" (UniqueName: \"kubernetes.io/projected/f937b72b-9f30-4465-bb35-f17afd3c5335-kube-api-access-h4dsb\") pod \"postgres-7cdbf6dc44-62qgf\" (UID: \"f937b72b-9f30-4465-bb35-f17afd3c5335\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"
Oct 30 14:00:34.530775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.530756    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f937b72b-9f30-4465-bb35-f17afd3c5335-data\") pod \"postgres-7cdbf6dc44-62qgf\" (UID: \"f937b72b-9f30-4465-bb35-f17afd3c5335\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"
Oct 30 14:00:34.559014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.558988    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4dsb\" (UniqueName: \"kubernetes.io/projected/f937b72b-9f30-4465-bb35-f17afd3c5335-kube-api-access-h4dsb\") pod \"postgres-7cdbf6dc44-62qgf\" (UID: \"f937b72b-9f30-4465-bb35-f17afd3c5335\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"
Oct 30 14:00:34.638662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.638609    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"
Oct 30 14:00:34.943844 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:00:34.943807    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf937b72b_9f30_4465_bb35_f17afd3c5335.slice/crio-0d2cc3fdb69ae9a65e37f20c11062547307b884a516971c9534bc8bf85f88e73 WatchSource:0}: Error finding container 0d2cc3fdb69ae9a65e37f20c11062547307b884a516971c9534bc8bf85f88e73: Status 404 returned error can't find the container with id 0d2cc3fdb69ae9a65e37f20c11062547307b884a516971c9534bc8bf85f88e73
Oct 30 14:00:34.944161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:34.944138    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"]
Oct 30 14:00:35.112912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:35.112874    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf" event={"ID":"f937b72b-9f30-4465-bb35-f17afd3c5335","Type":"ContainerStarted","Data":"0d2cc3fdb69ae9a65e37f20c11062547307b884a516971c9534bc8bf85f88e73"}
Oct 30 14:00:40.142296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:40.142215    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf" event={"ID":"f937b72b-9f30-4465-bb35-f17afd3c5335","Type":"ContainerStarted","Data":"9dbc65b6c14867da1b94862b708089e4089b89311e27746f78dba4f54ab35ea4"}
Oct 30 14:00:40.142687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:40.142299    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"
Oct 30 14:00:40.195111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:40.195065    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf" podStartSLOduration=1.274133287 podStartE2EDuration="6.195054057s" totalImagesPullingTime="4.92092077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:40.19406608 +0000 UTC m=+705.225965022" watchObservedRunningTime="2026-10-30 14:00:40.195054057 +0000 UTC m=+705.226952995"
Oct 30 14:00:41.365422 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:00:41.365388    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:00:46.175295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:00:46.175261    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-62qgf"
Oct 30 14:01:11.365103 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:01:11.365069    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:01:22.505096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:22.505058    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" podUID="e329b8d309e1f8141c44f91d4bb26de8" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 14:01:37.647610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:37.647573    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:01:41.366386 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:01:41.366349    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:01:47.460563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:47.460506    4530 generic.go:356] "Generic (PLEG): container finished" podID="b39d4963-4125-44ec-aa19-020b8e287167" containerID="fb0496343822c7dabae06ca9080c89d176c15a697b5459512fea8abd8fe8ea89" exitCode=0
Oct 30 14:01:47.460986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:47.460577    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" event={"ID":"b39d4963-4125-44ec-aa19-020b8e287167","Type":"ContainerDied","Data":"fb0496343822c7dabae06ca9080c89d176c15a697b5459512fea8abd8fe8ea89"}
Oct 30 14:01:47.460986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:47.460927    4530 scope.go:122] "RemoveContainer" containerID="fb0496343822c7dabae06ca9080c89d176c15a697b5459512fea8abd8fe8ea89"
Oct 30 14:01:47.461893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:47.461867    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:01:47.461949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:47.461930    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:01:47.461987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:47.461964    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:01:48.467161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:48.467123    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" event={"ID":"b39d4963-4125-44ec-aa19-020b8e287167","Type":"ContainerStarted","Data":"9b4d864a7f4b23e0cc628a8bcdb420d1a0be0454d30ecec53d2321139e92cc94"}
Oct 30 14:01:49.445954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:49.445910    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 14:01:49.447202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:49.447170    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-94bz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 14:01:49.447373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:49.447221    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" podUID="b39d4963-4125-44ec-aa19-020b8e287167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 14:01:50.446224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:50.446188    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-94bz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 14:01:50.446636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:50.446238    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" podUID="b39d4963-4125-44ec-aa19-020b8e287167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 14:01:50.888663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:50.888626    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 14:01:51.446387 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:51.446350    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-94bz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 14:01:51.446797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:51.446403    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" podUID="b39d4963-4125-44ec-aa19-020b8e287167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 14:01:52.446164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:52.446128    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-94bz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 14:01:52.446344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:52.446181    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" podUID="b39d4963-4125-44ec-aa19-020b8e287167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 14:01:53.466315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:53.466279    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 14:01:53.467485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:01:53.467457    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 14:02:11.365765 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:02:11.365722    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:02:37.090810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.090768    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"]
Oct 30 14:02:37.094481 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.094459    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.115299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.115270    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"]
Oct 30 14:02:37.124404 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.124381    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 14:02:37.124555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.124503    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lmhz8\""
Oct 30 14:02:37.125811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.125796    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 14:02:37.125913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.125900    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 14:02:37.175185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.175160    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7jj7\" (UniqueName: \"kubernetes.io/projected/6abb2398-ea14-431b-be41-1008dfd3fbf2-kube-api-access-x7jj7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.175282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.175197    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abb2398-ea14-431b-be41-1008dfd3fbf2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.175282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.175215    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abb2398-ea14-431b-be41-1008dfd3fbf2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.175354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.175287    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abb2398-ea14-431b-be41-1008dfd3fbf2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.175425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.175409    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abb2398-ea14-431b-be41-1008dfd3fbf2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.175479 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.175466    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abb2398-ea14-431b-be41-1008dfd3fbf2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.276863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.276835    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abb2398-ea14-431b-be41-1008dfd3fbf2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.277017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.276930    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abb2398-ea14-431b-be41-1008dfd3fbf2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.277017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.276968    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7jj7\" (UniqueName: \"kubernetes.io/projected/6abb2398-ea14-431b-be41-1008dfd3fbf2-kube-api-access-x7jj7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.277017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.277010    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abb2398-ea14-431b-be41-1008dfd3fbf2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.277170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.277032    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abb2398-ea14-431b-be41-1008dfd3fbf2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.277170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.277070    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abb2398-ea14-431b-be41-1008dfd3fbf2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.277306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.277282    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abb2398-ea14-431b-be41-1008dfd3fbf2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.277467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.277444    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abb2398-ea14-431b-be41-1008dfd3fbf2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.277467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.277373    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abb2398-ea14-431b-be41-1008dfd3fbf2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.279368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.279344    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abb2398-ea14-431b-be41-1008dfd3fbf2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.279554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.279521    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abb2398-ea14-431b-be41-1008dfd3fbf2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.305031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.305003    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7jj7\" (UniqueName: \"kubernetes.io/projected/6abb2398-ea14-431b-be41-1008dfd3fbf2-kube-api-access-x7jj7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx\" (UID: \"6abb2398-ea14-431b-be41-1008dfd3fbf2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.408508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.408447    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:37.708183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.708158    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"]
Oct 30 14:02:37.708799 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:02:37.708771    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abb2398_ea14_431b_be41_1008dfd3fbf2.slice/crio-acbaef4779b7057dbed47fc1b6079fece527cd1977eabde6183ef1710b7fe828 WatchSource:0}: Error finding container acbaef4779b7057dbed47fc1b6079fece527cd1977eabde6183ef1710b7fe828: Status 404 returned error can't find the container with id acbaef4779b7057dbed47fc1b6079fece527cd1977eabde6183ef1710b7fe828
Oct 30 14:02:37.729032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:37.728998    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx" event={"ID":"6abb2398-ea14-431b-be41-1008dfd3fbf2","Type":"ContainerStarted","Data":"acbaef4779b7057dbed47fc1b6079fece527cd1977eabde6183ef1710b7fe828"}
Oct 30 14:02:38.419596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:38.419555    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:02:39.740629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:39.740596    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx" event={"ID":"6abb2398-ea14-431b-be41-1008dfd3fbf2","Type":"ContainerStarted","Data":"b77b661778095026c9cd6fee30571ba24057b0afdbc3878a5731674423bb594e"}
Oct 30 14:02:40.408616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:40.408586    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:41.365296 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:02:41.365262    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:02:41.797424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:41.797374    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx" podStartSLOduration=3.719609779 podStartE2EDuration="4.797360642s" totalImagesPullingTime="1.077750863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:39.791932527 +0000 UTC m=+824.823831465" watchObservedRunningTime="2026-10-30 14:02:41.797360642 +0000 UTC m=+826.829259582"
Oct 30 14:02:41.799114 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:41.799093    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"]
Oct 30 14:02:41.802486 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:41.802464    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:41.835703 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:41.835679    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 14:02:41.846197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:41.846176    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"]
Oct 30 14:02:41.914985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:41.914964    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdbeeb1-2575-498d-9119-89c6714b3f51-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:41.915092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:41.915017    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccdbeeb1-2575-498d-9119-89c6714b3f51-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:41.915163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:41.915137    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccdbeeb1-2575-498d-9119-89c6714b3f51-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:41.915212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:41.915194    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccdbeeb1-2575-498d-9119-89c6714b3f51-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:41.915269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:41.915257    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccdbeeb1-2575-498d-9119-89c6714b3f51-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:41.915303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:41.915290    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkkrt\" (UniqueName: \"kubernetes.io/projected/ccdbeeb1-2575-498d-9119-89c6714b3f51-kube-api-access-zkkrt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.016472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.016443    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkrt\" (UniqueName: \"kubernetes.io/projected/ccdbeeb1-2575-498d-9119-89c6714b3f51-kube-api-access-zkkrt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.016651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.016490    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdbeeb1-2575-498d-9119-89c6714b3f51-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.016651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.016524    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccdbeeb1-2575-498d-9119-89c6714b3f51-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.016651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.016578    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccdbeeb1-2575-498d-9119-89c6714b3f51-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.016651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.016594    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccdbeeb1-2575-498d-9119-89c6714b3f51-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.016651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.016623    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccdbeeb1-2575-498d-9119-89c6714b3f51-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.017023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.017001    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccdbeeb1-2575-498d-9119-89c6714b3f51-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.017108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.017047    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccdbeeb1-2575-498d-9119-89c6714b3f51-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.017158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.017110    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccdbeeb1-2575-498d-9119-89c6714b3f51-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.018835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.018815    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccdbeeb1-2575-498d-9119-89c6714b3f51-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.019083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.019067    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdbeeb1-2575-498d-9119-89c6714b3f51-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.043619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.043596    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkrt\" (UniqueName: \"kubernetes.io/projected/ccdbeeb1-2575-498d-9119-89c6714b3f51-kube-api-access-zkkrt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zvnwp\" (UID: \"ccdbeeb1-2575-498d-9119-89c6714b3f51\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.114755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.114685    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:42.413893 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:02:42.413859    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdbeeb1_2575_498d_9119_89c6714b3f51.slice/crio-f4d3b49d5940416b1fa346250c8034e0d1251c8d9e6bc747f06515e8e5c6fbcc WatchSource:0}: Error finding container f4d3b49d5940416b1fa346250c8034e0d1251c8d9e6bc747f06515e8e5c6fbcc: Status 404 returned error can't find the container with id f4d3b49d5940416b1fa346250c8034e0d1251c8d9e6bc747f06515e8e5c6fbcc
Oct 30 14:02:42.414271 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.414079    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"]
Oct 30 14:02:42.757471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.757386    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp" event={"ID":"ccdbeeb1-2575-498d-9119-89c6714b3f51","Type":"ContainerStarted","Data":"adbe007265b75ac2e6131ac73d7b650995c7fbd175e275d53fe2088eae12d243"}
Oct 30 14:02:42.757471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.757445    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp" event={"ID":"ccdbeeb1-2575-498d-9119-89c6714b3f51","Type":"ContainerStarted","Data":"f4d3b49d5940416b1fa346250c8034e0d1251c8d9e6bc747f06515e8e5c6fbcc"}
Oct 30 14:02:42.811879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.811834    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp" podStartSLOduration=1.627657084 podStartE2EDuration="1.811820995s" totalImagesPullingTime="184.163911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:42.809847992 +0000 UTC m=+827.841746942" watchObservedRunningTime="2026-10-30 14:02:42.811820995 +0000 UTC m=+827.843719933"
Oct 30 14:02:42.907247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:42.907217    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:02:43.119803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:43.119771    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:45.043278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.043242    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"]
Oct 30 14:02:45.047640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.047616    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.065456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.065432    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"]
Oct 30 14:02:45.077216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.077196    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 14:02:45.142527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.142500    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.142642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.142534    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.142692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.142635    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.142692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.142674    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.142770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.142704    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.142770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.142759    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n8v2\" (UniqueName: \"kubernetes.io/projected/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-kube-api-access-8n8v2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.244106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.244066    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.244291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.244146    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.244291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.244176    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.244291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.244221    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.244291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.244253    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.244565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.244290    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8v2\" (UniqueName: \"kubernetes.io/projected/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-kube-api-access-8n8v2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.244628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.244588    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.244732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.244705    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.244818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.244798    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.246562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.246526    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.246769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.246741    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.270973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.270948    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8v2\" (UniqueName: \"kubernetes.io/projected/b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0-kube-api-access-8n8v2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v\" (UID: \"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.360411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.360390    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:45.663803 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:02:45.663774    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ab7ea0_dce1_44df_b4f6_af8e1564c7c0.slice/crio-877456c880c35cefd54ff0a0e310474a984f5e5dac39f1a7273c7c952ab6268a WatchSource:0}: Error finding container 877456c880c35cefd54ff0a0e310474a984f5e5dac39f1a7273c7c952ab6268a: Status 404 returned error can't find the container with id 877456c880c35cefd54ff0a0e310474a984f5e5dac39f1a7273c7c952ab6268a
Oct 30 14:02:45.664944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.664920    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"]
Oct 30 14:02:45.772781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:45.772751    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v" event={"ID":"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0","Type":"ContainerStarted","Data":"877456c880c35cefd54ff0a0e310474a984f5e5dac39f1a7273c7c952ab6268a"}
Oct 30 14:02:46.779093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:46.779050    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v" event={"ID":"b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0","Type":"ContainerStarted","Data":"bbf1fa75dd1ef51f4b975a180581ccd45b7ebf1ac51a0b8fd46a31dadf0075e5"}
Oct 30 14:02:46.827100 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:46.827053    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v" podStartSLOduration=1.645151848 podStartE2EDuration="1.827039487s" totalImagesPullingTime="181.887639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:46.825924304 +0000 UTC m=+831.857823243" watchObservedRunningTime="2026-10-30 14:02:46.827039487 +0000 UTC m=+831.858938425"
Oct 30 14:02:47.360795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:47.360768    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:47.409606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:47.409580    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:47.421876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:47.421853    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:47.665199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:47.665115    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:02:47.795992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:47.795962    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx"
Oct 30 14:02:52.115611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:52.115570    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:52.128111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:52.128084    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:52.823324 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:52.823294    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zvnwp"
Oct 30 14:02:54.781754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:54.781722    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:02:55.360557 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:55.360508    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:55.374953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:55.374929    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:02:55.847366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:02:55.847339    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v"
Oct 30 14:03:02.267600 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.267558    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"]
Oct 30 14:03:02.274560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.274523    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.302166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.302136    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"]
Oct 30 14:03:02.367331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.367302    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.367440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.367336    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.367440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.367369    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7t47\" (UniqueName: \"kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-kube-api-access-l7t47\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.367516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.367445    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.367516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.367472    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bfc568-8bae-43fe-adde-909edea76146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.367516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.367510    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.367691 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.367659    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.367745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.367700    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.367799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.367781    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bfc568-8bae-43fe-adde-909edea76146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.468179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.468152    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7t47\" (UniqueName: \"kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-kube-api-access-l7t47\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.468380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.468191    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.468380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.468207    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bfc568-8bae-43fe-adde-909edea76146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.468380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.468243    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.468380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.468264    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.468380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.468279    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.468380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.468315    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bfc568-8bae-43fe-adde-909edea76146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.468380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.468346    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.468380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.468370    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.468859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.468733    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.468859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.468815    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.469026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.469003    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.469149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.469129    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bfc568-8bae-43fe-adde-909edea76146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.469216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.469159    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.470499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.470479    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.470842 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.470823    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bfc568-8bae-43fe-adde-909edea76146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.496120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.496097    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.496350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.496327    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7t47\" (UniqueName: \"kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-kube-api-access-l7t47\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wstfm\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.621557 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.621522    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:02.941493 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:03:02.941463    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bfc568_8bae_43fe_adde_909edea76146.slice/crio-84fa087db8ccb07835fcf5bb2b83753bbcbb16e2191f0cbb359f283cf90d9a11 WatchSource:0}: Error finding container 84fa087db8ccb07835fcf5bb2b83753bbcbb16e2191f0cbb359f283cf90d9a11: Status 404 returned error can't find the container with id 84fa087db8ccb07835fcf5bb2b83753bbcbb16e2191f0cbb359f283cf90d9a11
Oct 30 14:03:02.942188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.942160    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"]
Oct 30 14:03:02.943361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.943326    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:03:02.943422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.943386    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:03:02.943422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:02.943416    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:03:03.906801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:03.906761    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" event={"ID":"c0bfc568-8bae-43fe-adde-909edea76146","Type":"ContainerStarted","Data":"ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5"}
Oct 30 14:03:03.907270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:03.906815    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" event={"ID":"c0bfc568-8bae-43fe-adde-909edea76146","Type":"ContainerStarted","Data":"84fa087db8ccb07835fcf5bb2b83753bbcbb16e2191f0cbb359f283cf90d9a11"}
Oct 30 14:03:03.958413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:03.958366    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" podStartSLOduration=1.9583529130000001 podStartE2EDuration="1.958352913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:03.957129407 +0000 UTC m=+848.989028345" watchObservedRunningTime="2026-10-30 14:03:03.958352913 +0000 UTC m=+848.990251851"
Oct 30 14:03:04.621948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:04.621907    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:04.623489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:04.623462    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wstfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 14:03:04.623626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:04.623509    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 14:03:05.438944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.438903    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"]
Oct 30 14:03:05.443049 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.443025    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"
Oct 30 14:03:05.463380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.463358    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"]
Oct 30 14:03:05.475775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.475743    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 14:03:05.475946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.475934    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mr58s\""
Oct 30 14:03:05.476029 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.475985    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 14:03:05.519167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.519125    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3604e30a-d71e-4ea0-be45-ade1d83e7d65-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-wg8lv\" (UID: \"3604e30a-d71e-4ea0-be45-ade1d83e7d65\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"
Oct 30 14:03:05.519321 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.519244    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h22fh\" (UniqueName: \"kubernetes.io/projected/3604e30a-d71e-4ea0-be45-ade1d83e7d65-kube-api-access-h22fh\") pod \"ai-gateway-controller-6cf7bcbfb6-wg8lv\" (UID: \"3604e30a-d71e-4ea0-be45-ade1d83e7d65\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"
Oct 30 14:03:05.619856 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.619814    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h22fh\" (UniqueName: \"kubernetes.io/projected/3604e30a-d71e-4ea0-be45-ade1d83e7d65-kube-api-access-h22fh\") pod \"ai-gateway-controller-6cf7bcbfb6-wg8lv\" (UID: \"3604e30a-d71e-4ea0-be45-ade1d83e7d65\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"
Oct 30 14:03:05.620065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.619957    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3604e30a-d71e-4ea0-be45-ade1d83e7d65-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-wg8lv\" (UID: \"3604e30a-d71e-4ea0-be45-ade1d83e7d65\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"
Oct 30 14:03:05.621856 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.621829    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wstfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 14:03:05.622611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.622588    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3604e30a-d71e-4ea0-be45-ade1d83e7d65-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-wg8lv\" (UID: \"3604e30a-d71e-4ea0-be45-ade1d83e7d65\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"
Oct 30 14:03:05.622962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.622914    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 14:03:05.647465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.647430    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h22fh\" (UniqueName: \"kubernetes.io/projected/3604e30a-d71e-4ea0-be45-ade1d83e7d65-kube-api-access-h22fh\") pod \"ai-gateway-controller-6cf7bcbfb6-wg8lv\" (UID: \"3604e30a-d71e-4ea0-be45-ade1d83e7d65\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"
Oct 30 14:03:05.759090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:05.759008    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"
Oct 30 14:03:06.101564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:06.099704    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"]
Oct 30 14:03:06.108245 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:03:06.108213    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3604e30a_d71e_4ea0_be45_ade1d83e7d65.slice/crio-c6335475c0212dc63c171c725549ce0570ca67ca8dce66168510ada35b3b09f2 WatchSource:0}: Error finding container c6335475c0212dc63c171c725549ce0570ca67ca8dce66168510ada35b3b09f2: Status 404 returned error can't find the container with id c6335475c0212dc63c171c725549ce0570ca67ca8dce66168510ada35b3b09f2
Oct 30 14:03:06.622672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:06.622637    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wstfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 14:03:06.623044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:06.622692    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 14:03:06.926421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:06.926283    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv" event={"ID":"3604e30a-d71e-4ea0-be45-ade1d83e7d65","Type":"ContainerStarted","Data":"c6335475c0212dc63c171c725549ce0570ca67ca8dce66168510ada35b3b09f2"}
Oct 30 14:03:07.622061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:07.622027    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wstfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 14:03:07.622254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:07.622076    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 14:03:08.622867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:08.622828    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wstfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 14:03:08.623354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:08.622889    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 14:03:09.622586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:09.622526    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wstfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 14:03:09.622763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:09.622602    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 14:03:09.951472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:09.951392    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv" event={"ID":"3604e30a-d71e-4ea0-be45-ade1d83e7d65","Type":"ContainerStarted","Data":"5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa"}
Oct 30 14:03:09.951870 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:09.951516    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"
Oct 30 14:03:10.000794 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.000746    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv" podStartSLOduration=2.013744185 podStartE2EDuration="5.000734337s" totalImagesPullingTime="2.986990152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:09.99834935 +0000 UTC m=+855.030248289" watchObservedRunningTime="2026-10-30 14:03:10.000734337 +0000 UTC m=+855.032633275"
Oct 30 14:03:10.421156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.421124    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"]
Oct 30 14:03:10.424636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.424618    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"
Oct 30 14:03:10.446180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.446158    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 14:03:10.446312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.446241    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-blhzn\""
Oct 30 14:03:10.446985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.446964    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"]
Oct 30 14:03:10.462814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.462770    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngnnk\" (UniqueName: \"kubernetes.io/projected/08de2a9c-9802-4b72-a844-58f650b6e573-kube-api-access-ngnnk\") pod \"payload-pre-processing-7b98bb678c-g99dv\" (UID: \"08de2a9c-9802-4b72-a844-58f650b6e573\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"
Oct 30 14:03:10.462944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.462820    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08de2a9c-9802-4b72-a844-58f650b6e573-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-g99dv\" (UID: \"08de2a9c-9802-4b72-a844-58f650b6e573\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"
Oct 30 14:03:10.486047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.486012    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-9dw4d"]
Oct 30 14:03:10.490254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.490231    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d"
Oct 30 14:03:10.501961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.501938    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-9dw4d"]
Oct 30 14:03:10.563408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.563369    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c340cff3-c8aa-40df-869a-5d5bf4c7210b-plugins-config-volume\") pod \"payload-processing-5f78687d5b-9dw4d\" (UID: \"c340cff3-c8aa-40df-869a-5d5bf4c7210b\") " pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d"
Oct 30 14:03:10.563630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.563551    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnnk\" (UniqueName: \"kubernetes.io/projected/08de2a9c-9802-4b72-a844-58f650b6e573-kube-api-access-ngnnk\") pod \"payload-pre-processing-7b98bb678c-g99dv\" (UID: \"08de2a9c-9802-4b72-a844-58f650b6e573\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"
Oct 30 14:03:10.563630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.563601    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqh79\" (UniqueName: \"kubernetes.io/projected/c340cff3-c8aa-40df-869a-5d5bf4c7210b-kube-api-access-rqh79\") pod \"payload-processing-5f78687d5b-9dw4d\" (UID: \"c340cff3-c8aa-40df-869a-5d5bf4c7210b\") " pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d"
Oct 30 14:03:10.563758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.563635    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08de2a9c-9802-4b72-a844-58f650b6e573-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-g99dv\" (UID: \"08de2a9c-9802-4b72-a844-58f650b6e573\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"
Oct 30 14:03:10.564326 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.564301    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08de2a9c-9802-4b72-a844-58f650b6e573-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-g99dv\" (UID: \"08de2a9c-9802-4b72-a844-58f650b6e573\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"
Oct 30 14:03:10.590673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.590649    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngnnk\" (UniqueName: \"kubernetes.io/projected/08de2a9c-9802-4b72-a844-58f650b6e573-kube-api-access-ngnnk\") pod \"payload-pre-processing-7b98bb678c-g99dv\" (UID: \"08de2a9c-9802-4b72-a844-58f650b6e573\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"
Oct 30 14:03:10.622099 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.622070    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wstfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 14:03:10.622219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.622122    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 14:03:10.665135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.665085    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqh79\" (UniqueName: \"kubernetes.io/projected/c340cff3-c8aa-40df-869a-5d5bf4c7210b-kube-api-access-rqh79\") pod \"payload-processing-5f78687d5b-9dw4d\" (UID: \"c340cff3-c8aa-40df-869a-5d5bf4c7210b\") " pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d"
Oct 30 14:03:10.665289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.665209    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c340cff3-c8aa-40df-869a-5d5bf4c7210b-plugins-config-volume\") pod \"payload-processing-5f78687d5b-9dw4d\" (UID: \"c340cff3-c8aa-40df-869a-5d5bf4c7210b\") " pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d"
Oct 30 14:03:10.665944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.665917    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c340cff3-c8aa-40df-869a-5d5bf4c7210b-plugins-config-volume\") pod \"payload-processing-5f78687d5b-9dw4d\" (UID: \"c340cff3-c8aa-40df-869a-5d5bf4c7210b\") " pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d"
Oct 30 14:03:10.691961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.691869    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqh79\" (UniqueName: \"kubernetes.io/projected/c340cff3-c8aa-40df-869a-5d5bf4c7210b-kube-api-access-rqh79\") pod \"payload-processing-5f78687d5b-9dw4d\" (UID: \"c340cff3-c8aa-40df-869a-5d5bf4c7210b\") " pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d"
Oct 30 14:03:10.737824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.737788    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"
Oct 30 14:03:10.808406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:10.808170    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d"
Oct 30 14:03:11.055833 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:03:11.055801    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08de2a9c_9802_4b72_a844_58f650b6e573.slice/crio-3dbb52a1d21418af3545db100a7db8189dd590717d788e9b0c583807054fdf16 WatchSource:0}: Error finding container 3dbb52a1d21418af3545db100a7db8189dd590717d788e9b0c583807054fdf16: Status 404 returned error can't find the container with id 3dbb52a1d21418af3545db100a7db8189dd590717d788e9b0c583807054fdf16
Oct 30 14:03:11.056349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:11.056331    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"]
Oct 30 14:03:11.135222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:11.135197    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-9dw4d"]
Oct 30 14:03:11.136213 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:03:11.136189    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc340cff3_c8aa_40df_869a_5d5bf4c7210b.slice/crio-314540f6eb4644652170cd530c08ef83d04ec0f9f99a2a1339f3c55f2d1a2b5e WatchSource:0}: Error finding container 314540f6eb4644652170cd530c08ef83d04ec0f9f99a2a1339f3c55f2d1a2b5e: Status 404 returned error can't find the container with id 314540f6eb4644652170cd530c08ef83d04ec0f9f99a2a1339f3c55f2d1a2b5e
Oct 30 14:03:11.368911 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:03:11.368818    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:03:11.967512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:11.966986    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv" event={"ID":"08de2a9c-9802-4b72-a844-58f650b6e573","Type":"ContainerStarted","Data":"3dbb52a1d21418af3545db100a7db8189dd590717d788e9b0c583807054fdf16"}
Oct 30 14:03:11.968762 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:11.968715    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d" event={"ID":"c340cff3-c8aa-40df-869a-5d5bf4c7210b","Type":"ContainerStarted","Data":"314540f6eb4644652170cd530c08ef83d04ec0f9f99a2a1339f3c55f2d1a2b5e"}
Oct 30 14:03:12.622712 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:12.622672    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wstfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:03:12.623188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:12.622738    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:03:13.663670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:13.663645    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wstfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:03:13.664042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:13.663687    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:03:13.990605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:13.990569    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv" event={"ID":"08de2a9c-9802-4b72-a844-58f650b6e573","Type":"ContainerStarted","Data":"3041c442e15cc4ba40b6401aa3aceb5e42053762f8429d366a9a80f4ca71df0c"}
Oct 30 14:03:13.990798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:13.990768    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"
Oct 30 14:03:13.992612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:13.992579    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d" event={"ID":"c340cff3-c8aa-40df-869a-5d5bf4c7210b","Type":"ContainerStarted","Data":"bfdab1f2cae6a88c052484d0ec6d14f6767f758596982a046d19c08c12c39429"}
Oct 30 14:03:13.992748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:13.992728    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d"
Oct 30 14:03:14.041033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:14.040964    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv" podStartSLOduration=1.516631121 podStartE2EDuration="4.040947331s" totalImagesPullingTime="2.52431621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:14.037272858 +0000 UTC m=+859.069171800" watchObservedRunningTime="2026-10-30 14:03:14.040947331 +0000 UTC m=+859.072846271"
Oct 30 14:03:14.086307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:14.086262    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d" podStartSLOduration=1.686949687 podStartE2EDuration="4.086249992s" totalImagesPullingTime="2.399300305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:14.083514327 +0000 UTC m=+859.115413267" watchObservedRunningTime="2026-10-30 14:03:14.086249992 +0000 UTC m=+859.118148976"
Oct 30 14:03:14.145067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:14.145041    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:14.145383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:14.145355    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:14.146025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:14.146009    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:03:20.960181 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:20.960154    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"
Oct 30 14:03:21.252224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:21.252154    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5bd6859554-ttz9m"]
Oct 30 14:03:21.255714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:21.255698    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m"
Oct 30 14:03:21.277140 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:21.277114    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5bd6859554-ttz9m"]
Oct 30 14:03:21.359758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:21.359734    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0490f682-51d0-4e20-a9cb-9cb1940ebb76-metrics-certs\") pod \"ai-gateway-controller-5bd6859554-ttz9m\" (UID: \"0490f682-51d0-4e20-a9cb-9cb1940ebb76\") " pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m"
Oct 30 14:03:21.359888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:21.359784    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgczz\" (UniqueName: \"kubernetes.io/projected/0490f682-51d0-4e20-a9cb-9cb1940ebb76-kube-api-access-sgczz\") pod \"ai-gateway-controller-5bd6859554-ttz9m\" (UID: \"0490f682-51d0-4e20-a9cb-9cb1940ebb76\") " pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m"
Oct 30 14:03:21.460981 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:21.460954    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0490f682-51d0-4e20-a9cb-9cb1940ebb76-metrics-certs\") pod \"ai-gateway-controller-5bd6859554-ttz9m\" (UID: \"0490f682-51d0-4e20-a9cb-9cb1940ebb76\") " pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m"
Oct 30 14:03:21.461083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:21.461000    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgczz\" (UniqueName: \"kubernetes.io/projected/0490f682-51d0-4e20-a9cb-9cb1940ebb76-kube-api-access-sgczz\") pod \"ai-gateway-controller-5bd6859554-ttz9m\" (UID: \"0490f682-51d0-4e20-a9cb-9cb1940ebb76\") " pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m"
Oct 30 14:03:21.463257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:21.463239    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0490f682-51d0-4e20-a9cb-9cb1940ebb76-metrics-certs\") pod \"ai-gateway-controller-5bd6859554-ttz9m\" (UID: \"0490f682-51d0-4e20-a9cb-9cb1940ebb76\") " pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m"
Oct 30 14:03:21.488462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:21.488433    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgczz\" (UniqueName: \"kubernetes.io/projected/0490f682-51d0-4e20-a9cb-9cb1940ebb76-kube-api-access-sgczz\") pod \"ai-gateway-controller-5bd6859554-ttz9m\" (UID: \"0490f682-51d0-4e20-a9cb-9cb1940ebb76\") " pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m"
Oct 30 14:03:21.567773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:21.567753    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m"
Oct 30 14:03:21.865354 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:03:21.865322    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0490f682_51d0_4e20_a9cb_9cb1940ebb76.slice/crio-0720a1e1c56b48db44598333298096a90192004d5adb2c4d6bc8cef981b83bde WatchSource:0}: Error finding container 0720a1e1c56b48db44598333298096a90192004d5adb2c4d6bc8cef981b83bde: Status 404 returned error can't find the container with id 0720a1e1c56b48db44598333298096a90192004d5adb2c4d6bc8cef981b83bde
Oct 30 14:03:21.865920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:21.865900    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5bd6859554-ttz9m"]
Oct 30 14:03:22.036368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:22.036340    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m" event={"ID":"0490f682-51d0-4e20-a9cb-9cb1940ebb76","Type":"ContainerStarted","Data":"0720a1e1c56b48db44598333298096a90192004d5adb2c4d6bc8cef981b83bde"}
Oct 30 14:03:23.044435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:23.044398    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m" event={"ID":"0490f682-51d0-4e20-a9cb-9cb1940ebb76","Type":"ContainerStarted","Data":"8fa502ec02a1a925418c502509b2e8fc779b8d9596bcdb7484d46655f3f702d1"}
Oct 30 14:03:23.044867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:23.044505    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m"
Oct 30 14:03:23.095081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:23.095039    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m" podStartSLOduration=1.734579525 podStartE2EDuration="2.095026829s" totalImagesPullingTime="360.447304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:23.094011309 +0000 UTC m=+868.125910245" watchObservedRunningTime="2026-10-30 14:03:23.095026829 +0000 UTC m=+868.126925767"
Oct 30 14:03:25.001921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:25.001891    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-g99dv"
Oct 30 14:03:25.002291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:25.001956    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-9dw4d"
Oct 30 14:03:34.051635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:34.051607    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5bd6859554-ttz9m"
Oct 30 14:03:34.132744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:34.132711    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"]
Oct 30 14:03:34.132939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:34.132881    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv" podUID="3604e30a-d71e-4ea0-be45-ade1d83e7d65" containerName="manager" containerID="cri-o://5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa" gracePeriod=30
Oct 30 14:03:34.384767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:34.384745    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"
Oct 30 14:03:34.468947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:34.468918    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3604e30a-d71e-4ea0-be45-ade1d83e7d65-kube-api-access-h22fh\" (UniqueName: \"kubernetes.io/projected/3604e30a-d71e-4ea0-be45-ade1d83e7d65-kube-api-access-h22fh\") pod \"3604e30a-d71e-4ea0-be45-ade1d83e7d65\" (UID: \"3604e30a-d71e-4ea0-be45-ade1d83e7d65\") "
Oct 30 14:03:34.469080 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:34.469050    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3604e30a-d71e-4ea0-be45-ade1d83e7d65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3604e30a-d71e-4ea0-be45-ade1d83e7d65-metrics-certs\") pod \"3604e30a-d71e-4ea0-be45-ade1d83e7d65\" (UID: \"3604e30a-d71e-4ea0-be45-ade1d83e7d65\") "
Oct 30 14:03:34.471059 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:34.471029    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3604e30a-d71e-4ea0-be45-ade1d83e7d65-kube-api-access-h22fh" pod "3604e30a-d71e-4ea0-be45-ade1d83e7d65" (UID: "3604e30a-d71e-4ea0-be45-ade1d83e7d65"). InnerVolumeSpecName "kube-api-access-h22fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:03:34.471059 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:34.471043    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3604e30a-d71e-4ea0-be45-ade1d83e7d65-metrics-certs" pod "3604e30a-d71e-4ea0-be45-ade1d83e7d65" (UID: "3604e30a-d71e-4ea0-be45-ade1d83e7d65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:03:34.570022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:34.569996    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3604e30a-d71e-4ea0-be45-ade1d83e7d65-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:03:34.570022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:34.570019    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h22fh\" (UniqueName: \"kubernetes.io/projected/3604e30a-d71e-4ea0-be45-ade1d83e7d65-kube-api-access-h22fh\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:03:35.111355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:35.111323    4530 generic.go:356] "Generic (PLEG): container finished" podID="3604e30a-d71e-4ea0-be45-ade1d83e7d65" containerID="5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa" exitCode=0
Oct 30 14:03:35.111783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:35.111408    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"
Oct 30 14:03:35.113108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:35.113085    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv" event={"ID":"3604e30a-d71e-4ea0-be45-ade1d83e7d65","Type":"ContainerDied","Data":"5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa"}
Oct 30 14:03:35.113195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:35.113126    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv" event={"ID":"3604e30a-d71e-4ea0-be45-ade1d83e7d65","Type":"ContainerDied","Data":"c6335475c0212dc63c171c725549ce0570ca67ca8dce66168510ada35b3b09f2"}
Oct 30 14:03:35.113195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:35.113153    4530 scope.go:122] "RemoveContainer" containerID="5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa"
Oct 30 14:03:35.123218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:35.123199    4530 scope.go:122] "RemoveContainer" containerID="5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa"
Oct 30 14:03:35.123470 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:03:35.123451    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa\": container with ID starting with 5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa not found: ID does not exist" containerID="5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa"
Oct 30 14:03:35.123526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:35.123479    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa"} err="failed to get container status \"5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa\": rpc error: code = NotFound desc = could not find container \"5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa\": container with ID starting with 5a8a80ff92e9347bc48e5967583ec8e383bf4bc81f85bd72537ab29d8bf50daa not found: ID does not exist"
Oct 30 14:03:35.171302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:35.171274    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"]
Oct 30 14:03:35.185120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:35.185095    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-wg8lv"]
Oct 30 14:03:37.113287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:37.113253    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3604e30a-d71e-4ea0-be45-ade1d83e7d65" path="/var/lib/kubelet/pods/3604e30a-d71e-4ea0-be45-ade1d83e7d65/volumes"
Oct 30 14:03:41.365828 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:03:41.365796    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:03:55.178098 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:55.178068    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:03:55.181222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:55.181200    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:03:55.183101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:55.183081    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:03:55.183946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:55.183923    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:03:55.186184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:55.186166    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:03:55.187084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:03:55.187065    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:04:11.365947 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:04:11.365909    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:04:23.485457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.485426    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"]
Oct 30 14:04:23.486075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.485888    4530 cpu_manager.go:414] "containerMap: removing container" podUID="3604e30a-d71e-4ea0-be45-ade1d83e7d65" containerName="manager"
Oct 30 14:04:23.486075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.485902    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3604e30a-d71e-4ea0-be45-ade1d83e7d65" containerName="manager"
Oct 30 14:04:23.486075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.485994    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="3604e30a-d71e-4ea0-be45-ade1d83e7d65" containerName="manager"
Oct 30 14:04:23.488174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.488154    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.514041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.514011    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"]
Oct 30 14:04:23.523637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.523614    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-00926c-openshift-default-dockercfg-2jkx6\""
Oct 30 14:04:23.570578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.570526    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.570776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.570752    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.570839 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.570816    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.570938 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.570916    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sp28\" (UniqueName: \"kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-kube-api-access-6sp28\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.570999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.570959    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97052caf-e138-4d01-8b20-a8cade8b7a3d-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.571080 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.571058    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.571138 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.571113    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.571236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.571211    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.571291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.571276    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.672055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672019    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.672678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672072    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sp28\" (UniqueName: \"kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-kube-api-access-6sp28\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.672678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672103    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97052caf-e138-4d01-8b20-a8cade8b7a3d-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.672678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672133    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.672678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672170    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.672678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672198    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.672678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672230    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.672678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672265    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.672678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672321    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.672678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672629    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.673332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672795    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.673332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672889    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.673332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.672922    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97052caf-e138-4d01-8b20-a8cade8b7a3d-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.673332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.673215    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.676247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.676198    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.677241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.677217    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.701107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.701080    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.701944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.701922    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sp28\" (UniqueName: \"kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-kube-api-access-6sp28\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.751943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.751876    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"]
Oct 30 14:04:23.758638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.758610    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.780081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.780058    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"]
Oct 30 14:04:23.780877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.780844    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a7bb0c-openshift-default-dockercfg-25sds\""
Oct 30 14:04:23.801790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.801758    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:23.879752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.878028    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.879752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.878136    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.879752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.878170    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.879752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.878236    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.879752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.878437    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfhhh\" (UniqueName: \"kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-kube-api-access-vfhhh\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.879752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.878518    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.879752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.878582    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.879752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.878612    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.879752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.878663    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.979745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.979716    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.979912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.979752    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.979912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.979775    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.979912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.979822    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfhhh\" (UniqueName: \"kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-kube-api-access-vfhhh\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.979912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.979853    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.979912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.979875    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.980143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.980037    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.980143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.980096    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.980246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.980201    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.980301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.980252    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.980357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.980314    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.980487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.980457    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.980654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.980589    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.980654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.980622    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.982117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.982098    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:23.982385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:23.982368    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:24.008469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.008416    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfhhh\" (UniqueName: \"kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-kube-api-access-vfhhh\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:24.011952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.011930    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:24.073790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.073759    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:24.143591 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.143330    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"]
Oct 30 14:04:24.145631 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:04:24.145601    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97052caf_e138_4d01_8b20_a8cade8b7a3d.slice/crio-b200e1a95183ab33fb3eaa99ffc0a37c222a0be4fb762a4ee531c1f185c1ca37 WatchSource:0}: Error finding container b200e1a95183ab33fb3eaa99ffc0a37c222a0be4fb762a4ee531c1f185c1ca37: Status 404 returned error can't find the container with id b200e1a95183ab33fb3eaa99ffc0a37c222a0be4fb762a4ee531c1f185c1ca37
Oct 30 14:04:24.148013 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.147973    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:24.148111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.148061    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:24.148156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.148107    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:24.189153 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.189128    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"]
Oct 30 14:04:24.193570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.193512    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.215480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.215459    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"]
Oct 30 14:04:24.245824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.245799    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-46b13b-openshift-default-dockercfg-922c6\""
Oct 30 14:04:24.284443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.284366    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da0e269-c913-4d18-8831-b9e89e184737-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.284443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.284405    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.284443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.284436    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da0e269-c913-4d18-8831-b9e89e184737-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.284716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.284519    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.284716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.284645    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.284716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.284709    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.284864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.284847    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.284929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.284909    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gq4j\" (UniqueName: \"kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-kube-api-access-2gq4j\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.284994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.284976    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.324988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.323381    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"]
Oct 30 14:04:24.326176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.326150    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.349438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.349232    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ca229c-openshift-default-dockercfg-52bwp\""
Oct 30 14:04:24.353470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.353434    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"]
Oct 30 14:04:24.366063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.366025    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" event={"ID":"97052caf-e138-4d01-8b20-a8cade8b7a3d","Type":"ContainerStarted","Data":"cedcf25d9c3d52ec664e14f379225288a89a2da59acd8bfe1c36c5b4c6a2f3b2"}
Oct 30 14:04:24.366184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.366076    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" event={"ID":"97052caf-e138-4d01-8b20-a8cade8b7a3d","Type":"ContainerStarted","Data":"b200e1a95183ab33fb3eaa99ffc0a37c222a0be4fb762a4ee531c1f185c1ca37"}
Oct 30 14:04:24.384819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.384798    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"]
Oct 30 14:04:24.386360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386337    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxxk2\" (UniqueName: \"kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-kube-api-access-qxxk2\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.386437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386393    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.386489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386457    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.386489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386477    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.386667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386561    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.386667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386633    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gq4j\" (UniqueName: \"kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-kube-api-access-2gq4j\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.386774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386655    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.386774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386700    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.386774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386721    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da0e269-c913-4d18-8831-b9e89e184737-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.386774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386742    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.386774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386767    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.387033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386790    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da0e269-c913-4d18-8831-b9e89e184737-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.387033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386808    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.387033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386828    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.387033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386864    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.387033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386886    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.387033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386898    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.387033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386910    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.387033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.386933    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.387493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.387471    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.387566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.387525    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.393593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.388082    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.393593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.388162    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da0e269-c913-4d18-8831-b9e89e184737-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.393593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.390578    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da0e269-c913-4d18-8831-b9e89e184737-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.394595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.394522    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:24.394701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.394607    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.394701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.394631    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:24.394701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.394672    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:24.414760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.414737    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gq4j\" (UniqueName: \"kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-kube-api-access-2gq4j\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.414873 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.414841    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.432060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.432023    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" podStartSLOduration=1.4320108 podStartE2EDuration="1.4320108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:24.431846219 +0000 UTC m=+929.463745171" watchObservedRunningTime="2026-10-30 14:04:24.4320108 +0000 UTC m=+929.463909738"
Oct 30 14:04:24.488456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.488409    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.488935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.488469    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.488935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.488502    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.488935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.488530    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.488935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.488654    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxxk2\" (UniqueName: \"kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-kube-api-access-qxxk2\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.488935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.488718    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.488935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.488750    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.488935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.488783    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.488935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.488863    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.488935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.488891    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.489504 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.489471    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.489789 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.489763    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.490223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.490200    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.490392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.490370    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.492064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.492012    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.493136 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.493113    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.515492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.515469    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxxk2\" (UniqueName: \"kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-kube-api-access-qxxk2\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.515725 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.515704    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.518636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.518604    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:24.640034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.639753    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:24.802497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.802459    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:24.872686 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:04:24.872654    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da0e269_c913_4d18_8831_b9e89e184737.slice/crio-28812084eb156cb486311cfa1b246ab658a84cbb06aefb95be291d1c36ef800b WatchSource:0}: Error finding container 28812084eb156cb486311cfa1b246ab658a84cbb06aefb95be291d1c36ef800b: Status 404 returned error can't find the container with id 28812084eb156cb486311cfa1b246ab658a84cbb06aefb95be291d1c36ef800b
Oct 30 14:04:24.873286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.873258    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"]
Oct 30 14:04:24.875121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.875073    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:24.875227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.875166    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:24.875227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.875210    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:24.961800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.961771    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"]
Oct 30 14:04:24.966449 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:04:24.966422    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2e3f8d_e484_4ec1_9516_d7d34fef7c0a.slice/crio-814c08df4e961853edc4092428edaff788cc5d5315afe5f5a22c1edd87e5512b WatchSource:0}: Error finding container 814c08df4e961853edc4092428edaff788cc5d5315afe5f5a22c1edd87e5512b: Status 404 returned error can't find the container with id 814c08df4e961853edc4092428edaff788cc5d5315afe5f5a22c1edd87e5512b
Oct 30 14:04:24.969287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.969246    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:24.969423 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.969331    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:24.969423 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:24.969373    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:25.374417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.374379    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" event={"ID":"9da0e269-c913-4d18-8831-b9e89e184737","Type":"ContainerStarted","Data":"420afa151307904bbc62416a306c9a02e3604e2529c4cfeba67812c50ed01035"}
Oct 30 14:04:25.374615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.374434    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" event={"ID":"9da0e269-c913-4d18-8831-b9e89e184737","Type":"ContainerStarted","Data":"28812084eb156cb486311cfa1b246ab658a84cbb06aefb95be291d1c36ef800b"}
Oct 30 14:04:25.376188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.376157    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" event={"ID":"9b5c1a9e-e28f-4501-ab13-a4099e4220d3","Type":"ContainerStarted","Data":"963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b"}
Oct 30 14:04:25.376314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.376210    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" event={"ID":"9b5c1a9e-e28f-4501-ab13-a4099e4220d3","Type":"ContainerStarted","Data":"c12b797e96a6411616061c4819367af74b81c8a301e407d348c48e39a62d5e2e"}
Oct 30 14:04:25.377975 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.377947    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" event={"ID":"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a","Type":"ContainerStarted","Data":"0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873"}
Oct 30 14:04:25.378080 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.378006    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" event={"ID":"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a","Type":"ContainerStarted","Data":"814c08df4e961853edc4092428edaff788cc5d5315afe5f5a22c1edd87e5512b"}
Oct 30 14:04:25.432253 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.432192    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" podStartSLOduration=1.4321745350000001 podStartE2EDuration="1.432174535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:25.430638064 +0000 UTC m=+930.462537007" watchObservedRunningTime="2026-10-30 14:04:25.432174535 +0000 UTC m=+930.464073474"
Oct 30 14:04:25.483160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.483108    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" podStartSLOduration=1.483093689 podStartE2EDuration="1.483093689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:25.480605911 +0000 UTC m=+930.512504853" watchObservedRunningTime="2026-10-30 14:04:25.483093689 +0000 UTC m=+930.514992628"
Oct 30 14:04:25.519231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.519205    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:25.524084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.524061    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:25.532052 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.532015    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" podStartSLOduration=2.532003235 podStartE2EDuration="2.532003235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:25.530386281 +0000 UTC m=+930.562285235" watchObservedRunningTime="2026-10-30 14:04:25.532003235 +0000 UTC m=+930.563902174"
Oct 30 14:04:25.646566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.644096    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:25.807359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:25.807329    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:26.074724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:26.074698    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:26.079449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:26.079424    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:26.387848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:26.387768    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:26.387848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:26.387828    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:26.388087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:26.387855    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:26.388416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:26.388380    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:04:26.388572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:26.388531    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:04:26.388732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:26.388715    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:26.645815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:26.645743    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:27.398199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:27.398168    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:27.398566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:27.398522    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:04:29.602196 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.602165    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"]
Oct 30 14:04:29.604836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.604819    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"
Oct 30 14:04:29.628126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.628090    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"]
Oct 30 14:04:29.638472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.638448    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-00926c\""
Oct 30 14:04:29.638597 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.638526    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-00926c-dockercfg-rwhkw\""
Oct 30 14:04:29.696383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.696342    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"]
Oct 30 14:04:29.715060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.715027    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"]
Oct 30 14:04:29.715234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.715139    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"
Oct 30 14:04:29.738697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.738668    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9lv\" (UniqueName: \"kubernetes.io/projected/da680b84-e339-4baf-adfb-e6519af3fbbd-kube-api-access-6m9lv\") pod \"payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z\" (UID: \"da680b84-e339-4baf-adfb-e6519af3fbbd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"
Oct 30 14:04:29.738840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.738768    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da680b84-e339-4baf-adfb-e6519af3fbbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z\" (UID: \"da680b84-e339-4baf-adfb-e6519af3fbbd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"
Oct 30 14:04:29.839948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.839912    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da680b84-e339-4baf-adfb-e6519af3fbbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z\" (UID: \"da680b84-e339-4baf-adfb-e6519af3fbbd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"
Oct 30 14:04:29.840112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.840012    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcpx2\" (UniqueName: \"kubernetes.io/projected/ef6239cf-ec71-4299-8b6d-ef31d72c3162-kube-api-access-rcpx2\") pod \"payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp\" (UID: \"ef6239cf-ec71-4299-8b6d-ef31d72c3162\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"
Oct 30 14:04:29.840112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.840046    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9lv\" (UniqueName: \"kubernetes.io/projected/da680b84-e339-4baf-adfb-e6519af3fbbd-kube-api-access-6m9lv\") pod \"payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z\" (UID: \"da680b84-e339-4baf-adfb-e6519af3fbbd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"
Oct 30 14:04:29.840206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.840122    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6239cf-ec71-4299-8b6d-ef31d72c3162-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp\" (UID: \"ef6239cf-ec71-4299-8b6d-ef31d72c3162\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"
Oct 30 14:04:29.840635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.840615    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da680b84-e339-4baf-adfb-e6519af3fbbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z\" (UID: \"da680b84-e339-4baf-adfb-e6519af3fbbd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"
Oct 30 14:04:29.866333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.866274    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9lv\" (UniqueName: \"kubernetes.io/projected/da680b84-e339-4baf-adfb-e6519af3fbbd-kube-api-access-6m9lv\") pod \"payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z\" (UID: \"da680b84-e339-4baf-adfb-e6519af3fbbd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"
Oct 30 14:04:29.916527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.916500    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"
Oct 30 14:04:29.941399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.941373    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6239cf-ec71-4299-8b6d-ef31d72c3162-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp\" (UID: \"ef6239cf-ec71-4299-8b6d-ef31d72c3162\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"
Oct 30 14:04:29.941511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.941495    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcpx2\" (UniqueName: \"kubernetes.io/projected/ef6239cf-ec71-4299-8b6d-ef31d72c3162-kube-api-access-rcpx2\") pod \"payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp\" (UID: \"ef6239cf-ec71-4299-8b6d-ef31d72c3162\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"
Oct 30 14:04:29.942119 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.942100    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6239cf-ec71-4299-8b6d-ef31d72c3162-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp\" (UID: \"ef6239cf-ec71-4299-8b6d-ef31d72c3162\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"
Oct 30 14:04:29.970175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:29.970151    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcpx2\" (UniqueName: \"kubernetes.io/projected/ef6239cf-ec71-4299-8b6d-ef31d72c3162-kube-api-access-rcpx2\") pod \"payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp\" (UID: \"ef6239cf-ec71-4299-8b6d-ef31d72c3162\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"
Oct 30 14:04:30.032867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:30.032835    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"
Oct 30 14:04:30.212285 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:04:30.212250    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda680b84_e339_4baf_adfb_e6519af3fbbd.slice/crio-a4835b82c64e830d8e0e317999e63f24714159cc4da7df25058d24807adb78ed WatchSource:0}: Error finding container a4835b82c64e830d8e0e317999e63f24714159cc4da7df25058d24807adb78ed: Status 404 returned error can't find the container with id a4835b82c64e830d8e0e317999e63f24714159cc4da7df25058d24807adb78ed
Oct 30 14:04:30.212843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:30.212820    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"]
Oct 30 14:04:30.332089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:30.332065    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"]
Oct 30 14:04:30.332174 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:04:30.332148    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6239cf_ec71_4299_8b6d_ef31d72c3162.slice/crio-aace5e15d468166e31135e936776cd350e21cdda83f91440982e12e858c630b6 WatchSource:0}: Error finding container aace5e15d468166e31135e936776cd350e21cdda83f91440982e12e858c630b6: Status 404 returned error can't find the container with id aace5e15d468166e31135e936776cd350e21cdda83f91440982e12e858c630b6
Oct 30 14:04:30.412473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:30.412442    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z" event={"ID":"da680b84-e339-4baf-adfb-e6519af3fbbd","Type":"ContainerStarted","Data":"05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597"}
Oct 30 14:04:30.412681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:30.412652    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"
Oct 30 14:04:30.412793 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:30.412703    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z" event={"ID":"da680b84-e339-4baf-adfb-e6519af3fbbd","Type":"ContainerStarted","Data":"a4835b82c64e830d8e0e317999e63f24714159cc4da7df25058d24807adb78ed"}
Oct 30 14:04:30.414091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:30.414066    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp" event={"ID":"ef6239cf-ec71-4299-8b6d-ef31d72c3162","Type":"ContainerStarted","Data":"7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40"}
Oct 30 14:04:30.414192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:30.414127    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp" event={"ID":"ef6239cf-ec71-4299-8b6d-ef31d72c3162","Type":"ContainerStarted","Data":"aace5e15d468166e31135e936776cd350e21cdda83f91440982e12e858c630b6"}
Oct 30 14:04:30.414192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:30.414160    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"
Oct 30 14:04:30.465933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:30.465857    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z" podStartSLOduration=1.465845825 podStartE2EDuration="1.465845825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:30.46319182 +0000 UTC m=+935.495090759" watchObservedRunningTime="2026-10-30 14:04:30.465845825 +0000 UTC m=+935.497744820"
Oct 30 14:04:30.508381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:30.508346    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp" podStartSLOduration=1.508334136 podStartE2EDuration="1.508334136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:30.508077881 +0000 UTC m=+935.539976824" watchObservedRunningTime="2026-10-30 14:04:30.508334136 +0000 UTC m=+935.540233069"
Oct 30 14:04:33.037090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.037057    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"]
Oct 30 14:04:33.040224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.040202    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"
Oct 30 14:04:33.060986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.060962    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"]
Oct 30 14:04:33.071687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.071665    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2df0ab\""
Oct 30 14:04:33.071838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.071774    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2df0ab-dockercfg-994gs\""
Oct 30 14:04:33.180460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.180425    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ba7a80-b30b-40b3-a209-674a54c6f096-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj\" (UID: \"f4ba7a80-b30b-40b3-a209-674a54c6f096\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"
Oct 30 14:04:33.180627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.180560    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzjjr\" (UniqueName: \"kubernetes.io/projected/f4ba7a80-b30b-40b3-a209-674a54c6f096-kube-api-access-wzjjr\") pod \"payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj\" (UID: \"f4ba7a80-b30b-40b3-a209-674a54c6f096\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"
Oct 30 14:04:33.281728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.281696    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ba7a80-b30b-40b3-a209-674a54c6f096-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj\" (UID: \"f4ba7a80-b30b-40b3-a209-674a54c6f096\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"
Oct 30 14:04:33.281907 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.281782    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzjjr\" (UniqueName: \"kubernetes.io/projected/f4ba7a80-b30b-40b3-a209-674a54c6f096-kube-api-access-wzjjr\") pod \"payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj\" (UID: \"f4ba7a80-b30b-40b3-a209-674a54c6f096\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"
Oct 30 14:04:33.282484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.282465    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ba7a80-b30b-40b3-a209-674a54c6f096-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj\" (UID: \"f4ba7a80-b30b-40b3-a209-674a54c6f096\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"
Oct 30 14:04:33.308765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.308734    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzjjr\" (UniqueName: \"kubernetes.io/projected/f4ba7a80-b30b-40b3-a209-674a54c6f096-kube-api-access-wzjjr\") pod \"payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj\" (UID: \"f4ba7a80-b30b-40b3-a209-674a54c6f096\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"
Oct 30 14:04:33.352672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.352643    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"
Oct 30 14:04:33.652150 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:04:33.652113    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ba7a80_b30b_40b3_a209_674a54c6f096.slice/crio-ae594ef12aa0a1588ba74b74d807692ce6682fd22c76be17b72c0a250583e672 WatchSource:0}: Error finding container ae594ef12aa0a1588ba74b74d807692ce6682fd22c76be17b72c0a250583e672: Status 404 returned error can't find the container with id ae594ef12aa0a1588ba74b74d807692ce6682fd22c76be17b72c0a250583e672
Oct 30 14:04:33.652344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:33.652320    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"]
Oct 30 14:04:34.439888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:34.439852    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj" event={"ID":"f4ba7a80-b30b-40b3-a209-674a54c6f096","Type":"ContainerStarted","Data":"9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798"}
Oct 30 14:04:34.440275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:34.439992    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"
Oct 30 14:04:34.440275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:34.440011    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj" event={"ID":"f4ba7a80-b30b-40b3-a209-674a54c6f096","Type":"ContainerStarted","Data":"ae594ef12aa0a1588ba74b74d807692ce6682fd22c76be17b72c0a250583e672"}
Oct 30 14:04:34.486012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:34.485969    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj" podStartSLOduration=1.485956447 podStartE2EDuration="1.485956447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:34.484921316 +0000 UTC m=+939.516820257" watchObservedRunningTime="2026-10-30 14:04:34.485956447 +0000 UTC m=+939.517855386"
Oct 30 14:04:35.012732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.012703    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"]
Oct 30 14:04:35.015683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.015669    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.036658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.036604    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"]
Oct 30 14:04:35.046955 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.046925    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-da78c4-serving-cert\""
Oct 30 14:04:35.047106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.046998    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 14:04:35.047106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.047091    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-54whp\""
Oct 30 14:04:35.100212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.100181    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.100417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.100394    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h7sb\" (UniqueName: \"kubernetes.io/projected/85899396-118d-4899-a445-2077f6778fc0-kube-api-access-7h7sb\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.100464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.100434    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-metrics-certs\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.100508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.100493    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85899396-118d-4899-a445-2077f6778fc0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.201615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.201581    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.201777 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:04:35.201757    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-serving-cert: secret "maas-api-e2e-worker-w2-da78c4-serving-cert" not found
Oct 30 14:04:35.201844 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:04:35.201829    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-maas-api-tls podName:85899396-118d-4899-a445-2077f6778fc0 nodeName:}" failed. No retries permitted until 2026-10-30 14:04:35.701807837 +0000 UTC m=+940.733706770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-maas-api-tls") pod "maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95" (UID: "85899396-118d-4899-a445-2077f6778fc0") : secret "maas-api-e2e-worker-w2-da78c4-serving-cert" not found
Oct 30 14:04:35.201933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.201890    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h7sb\" (UniqueName: \"kubernetes.io/projected/85899396-118d-4899-a445-2077f6778fc0-kube-api-access-7h7sb\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.201985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.201931    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-metrics-certs\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.201985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.201976    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85899396-118d-4899-a445-2077f6778fc0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.202592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.202575    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85899396-118d-4899-a445-2077f6778fc0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.204168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.204148    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-metrics-certs\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.231076 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.231045    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h7sb\" (UniqueName: \"kubernetes.io/projected/85899396-118d-4899-a445-2077f6778fc0-kube-api-access-7h7sb\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.337855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.337826    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"]
Oct 30 14:04:35.340512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.340496    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"
Oct 30 14:04:35.360040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.360012    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"]
Oct 30 14:04:35.403351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.403326    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dtk5\" (UniqueName: \"kubernetes.io/projected/9c668c61-9ca2-4958-a9cc-e1d274b422a7-kube-api-access-7dtk5\") pod \"payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj\" (UID: \"9c668c61-9ca2-4958-a9cc-e1d274b422a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"
Oct 30 14:04:35.403487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.403422    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c668c61-9ca2-4958-a9cc-e1d274b422a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj\" (UID: \"9c668c61-9ca2-4958-a9cc-e1d274b422a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"
Oct 30 14:04:35.418305 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.418287    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-a7bb0c\""
Oct 30 14:04:35.512203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.512165    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c668c61-9ca2-4958-a9cc-e1d274b422a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj\" (UID: \"9c668c61-9ca2-4958-a9cc-e1d274b422a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"
Oct 30 14:04:35.512782 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.512406    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dtk5\" (UniqueName: \"kubernetes.io/projected/9c668c61-9ca2-4958-a9cc-e1d274b422a7-kube-api-access-7dtk5\") pod \"payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj\" (UID: \"9c668c61-9ca2-4958-a9cc-e1d274b422a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"
Oct 30 14:04:35.514707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.513699    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c668c61-9ca2-4958-a9cc-e1d274b422a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj\" (UID: \"9c668c61-9ca2-4958-a9cc-e1d274b422a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"
Oct 30 14:04:35.539473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.539444    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dtk5\" (UniqueName: \"kubernetes.io/projected/9c668c61-9ca2-4958-a9cc-e1d274b422a7-kube-api-access-7dtk5\") pod \"payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj\" (UID: \"9c668c61-9ca2-4958-a9cc-e1d274b422a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"
Oct 30 14:04:35.653158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.653086    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"
Oct 30 14:04:35.715663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.715570    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.718159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.718135    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.927341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.927277    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:35.954531 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:04:35.954147    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c668c61_9ca2_4958_a9cc_e1d274b422a7.slice/crio-e7e5b5c12d0d79d53320d52dd60f32a8337263658af69220081d7264f0e8e8c1 WatchSource:0}: Error finding container e7e5b5c12d0d79d53320d52dd60f32a8337263658af69220081d7264f0e8e8c1: Status 404 returned error can't find the container with id e7e5b5c12d0d79d53320d52dd60f32a8337263658af69220081d7264f0e8e8c1
Oct 30 14:04:35.954531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:35.954322    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"]
Oct 30 14:04:36.232833 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:04:36.232805    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85899396_118d_4899_a445_2077f6778fc0.slice/crio-0fd3a2efd0936f732b14fc2345c23080cf44fba08d9259489c18f6187cfd7a1b WatchSource:0}: Error finding container 0fd3a2efd0936f732b14fc2345c23080cf44fba08d9259489c18f6187cfd7a1b: Status 404 returned error can't find the container with id 0fd3a2efd0936f732b14fc2345c23080cf44fba08d9259489c18f6187cfd7a1b
Oct 30 14:04:36.234250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:36.234224    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"]
Oct 30 14:04:36.453646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:36.453602    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj" event={"ID":"9c668c61-9ca2-4958-a9cc-e1d274b422a7","Type":"ContainerStarted","Data":"5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d"}
Oct 30 14:04:36.453827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:36.453666    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj" event={"ID":"9c668c61-9ca2-4958-a9cc-e1d274b422a7","Type":"ContainerStarted","Data":"e7e5b5c12d0d79d53320d52dd60f32a8337263658af69220081d7264f0e8e8c1"}
Oct 30 14:04:36.453827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:36.453798    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"
Oct 30 14:04:36.454890 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:36.454867    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95" event={"ID":"85899396-118d-4899-a445-2077f6778fc0","Type":"ContainerStarted","Data":"0fd3a2efd0936f732b14fc2345c23080cf44fba08d9259489c18f6187cfd7a1b"}
Oct 30 14:04:36.500496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:36.500425    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj" podStartSLOduration=1.500414794 podStartE2EDuration="1.500414794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:36.500163222 +0000 UTC m=+941.532062173" watchObservedRunningTime="2026-10-30 14:04:36.500414794 +0000 UTC m=+941.532313732"
Oct 30 14:04:36.579113 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:36.579088    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:04:36.940493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:36.940464    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"]
Oct 30 14:04:36.952918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:36.952882    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"
Oct 30 14:04:36.965067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:36.964782    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"]
Oct 30 14:04:36.972763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:36.971174    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-46b13b\""
Oct 30 14:04:37.031500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:37.031323    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndbc5\" (UniqueName: \"kubernetes.io/projected/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-kube-api-access-ndbc5\") pod \"payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2\" (UID: \"e54a8e55-53ec-4a30-a4ec-bc39634e5a63\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"
Oct 30 14:04:37.031500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:37.031381    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2\" (UID: \"e54a8e55-53ec-4a30-a4ec-bc39634e5a63\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"
Oct 30 14:04:37.135284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:37.135230    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndbc5\" (UniqueName: \"kubernetes.io/projected/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-kube-api-access-ndbc5\") pod \"payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2\" (UID: \"e54a8e55-53ec-4a30-a4ec-bc39634e5a63\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"
Oct 30 14:04:37.135421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:37.135312    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2\" (UID: \"e54a8e55-53ec-4a30-a4ec-bc39634e5a63\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"
Oct 30 14:04:37.138595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:37.137773    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2\" (UID: \"e54a8e55-53ec-4a30-a4ec-bc39634e5a63\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"
Oct 30 14:04:37.175410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:37.175375    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndbc5\" (UniqueName: \"kubernetes.io/projected/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-kube-api-access-ndbc5\") pod \"payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2\" (UID: \"e54a8e55-53ec-4a30-a4ec-bc39634e5a63\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"
Oct 30 14:04:37.272754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:37.272679    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"
Oct 30 14:04:37.612243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:37.612211    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"]
Oct 30 14:04:37.615445 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:04:37.615415    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54a8e55_53ec_4a30_a4ec_bc39634e5a63.slice/crio-138f2fdda6b8899f5619d906033e233ee2b68ba49f6601a4c7be2226c8e64c56 WatchSource:0}: Error finding container 138f2fdda6b8899f5619d906033e233ee2b68ba49f6601a4c7be2226c8e64c56: Status 404 returned error can't find the container with id 138f2fdda6b8899f5619d906033e233ee2b68ba49f6601a4c7be2226c8e64c56
Oct 30 14:04:38.474806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:38.474761    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2" event={"ID":"e54a8e55-53ec-4a30-a4ec-bc39634e5a63","Type":"ContainerStarted","Data":"ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2"}
Oct 30 14:04:38.475005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:38.474820    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2" event={"ID":"e54a8e55-53ec-4a30-a4ec-bc39634e5a63","Type":"ContainerStarted","Data":"138f2fdda6b8899f5619d906033e233ee2b68ba49f6601a4c7be2226c8e64c56"}
Oct 30 14:04:38.475005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:38.474910    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"
Oct 30 14:04:38.533418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:38.533364    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2" podStartSLOduration=2.533346872 podStartE2EDuration="2.533346872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:38.531941906 +0000 UTC m=+943.563840848" watchObservedRunningTime="2026-10-30 14:04:38.533346872 +0000 UTC m=+943.565245813"
Oct 30 14:04:39.505949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:39.505916    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95" event={"ID":"85899396-118d-4899-a445-2077f6778fc0","Type":"ContainerStarted","Data":"910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644"}
Oct 30 14:04:39.506403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:39.506267    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:39.554372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:39.554326    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95" podStartSLOduration=2.46453281 podStartE2EDuration="5.55431355s" totalImagesPullingTime="3.08978074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:39.552774719 +0000 UTC m=+944.584673658" watchObservedRunningTime="2026-10-30 14:04:39.55431355 +0000 UTC m=+944.586212489"
Oct 30 14:04:41.305861 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:41.305815    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:04:41.374393 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:04:41.374354    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:04:41.423591 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:41.423564    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"
Oct 30 14:04:41.423721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:41.423627    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"
Oct 30 14:04:45.429751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.429719    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"]
Oct 30 14:04:45.432696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.432680    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.449601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.449579    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"
Oct 30 14:04:45.471946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.471897    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-46b13b-serving-cert\""
Oct 30 14:04:45.474963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.474932    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"]
Oct 30 14:04:45.518844 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.518815    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:04:45.555583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.555515    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbhm\" (UniqueName: \"kubernetes.io/projected/4545aeaf-ba17-4323-9e81-86d345f50d6b-kube-api-access-vkbhm\") pod \"maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.556214 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.556181    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.556338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.556308    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-metrics-certs\") pod \"maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.556401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.556363    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4545aeaf-ba17-4323-9e81-86d345f50d6b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.657825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.657776    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbhm\" (UniqueName: \"kubernetes.io/projected/4545aeaf-ba17-4323-9e81-86d345f50d6b-kube-api-access-vkbhm\") pod \"maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.658003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.657832    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.658003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.657895    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-metrics-certs\") pod \"maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.658003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.657959    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4545aeaf-ba17-4323-9e81-86d345f50d6b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.658870 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.658832    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4545aeaf-ba17-4323-9e81-86d345f50d6b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.660482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.660454    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.660702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.660675    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-metrics-certs\") pod \"maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.685204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.685138    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbhm\" (UniqueName: \"kubernetes.io/projected/4545aeaf-ba17-4323-9e81-86d345f50d6b-kube-api-access-vkbhm\") pod \"maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:45.745571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:45.745527    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:46.058753 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:04:46.058726    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4545aeaf_ba17_4323_9e81_86d345f50d6b.slice/crio-e9d74b1e5685fb8bbbe9b78fd8442ded3cd33040f3539e7355c3cced7246f972 WatchSource:0}: Error finding container e9d74b1e5685fb8bbbe9b78fd8442ded3cd33040f3539e7355c3cced7246f972: Status 404 returned error can't find the container with id e9d74b1e5685fb8bbbe9b78fd8442ded3cd33040f3539e7355c3cced7246f972
Oct 30 14:04:46.059166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:46.059140    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"]
Oct 30 14:04:46.574577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:46.574524    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:04:46.581448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:46.581263    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m" event={"ID":"4545aeaf-ba17-4323-9e81-86d345f50d6b","Type":"ContainerStarted","Data":"56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b"}
Oct 30 14:04:46.581448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:46.581335    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m" event={"ID":"4545aeaf-ba17-4323-9e81-86d345f50d6b","Type":"ContainerStarted","Data":"e9d74b1e5685fb8bbbe9b78fd8442ded3cd33040f3539e7355c3cced7246f972"}
Oct 30 14:04:46.581448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:46.581402    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:46.583637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:46.583611    4530 generic.go:356] "Generic (PLEG): container finished" podID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerID="cedcf25d9c3d52ec664e14f379225288a89a2da59acd8bfe1c36c5b4c6a2f3b2" exitCode=0
Oct 30 14:04:46.583760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:46.583665    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" event={"ID":"97052caf-e138-4d01-8b20-a8cade8b7a3d","Type":"ContainerDied","Data":"cedcf25d9c3d52ec664e14f379225288a89a2da59acd8bfe1c36c5b4c6a2f3b2"}
Oct 30 14:04:46.584151 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:46.584137    4530 scope.go:122] "RemoveContainer" containerID="cedcf25d9c3d52ec664e14f379225288a89a2da59acd8bfe1c36c5b4c6a2f3b2"
Oct 30 14:04:46.585460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:46.585435    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:46.585631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:46.585518    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:46.585722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:46.585675    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:46.667556 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:46.667460    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m" podStartSLOduration=1.326965733 podStartE2EDuration="1.667447415s" totalImagesPullingTime="340.481682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:46.665283623 +0000 UTC m=+951.697182560" watchObservedRunningTime="2026-10-30 14:04:46.667447415 +0000 UTC m=+951.699346353"
Oct 30 14:04:47.465496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:47.465464    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"
Oct 30 14:04:47.619584 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:47.618667    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" event={"ID":"97052caf-e138-4d01-8b20-a8cade8b7a3d","Type":"ContainerStarted","Data":"9e1028d0d75b9cdb5ed3208f264c354558f5ca8ac8b8192e29e8f6f126a37f9f"}
Oct 30 14:04:47.805626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:47.805234    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:47.805626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:47.805350    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 14:04:47.805626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:47.805392    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 14:04:48.813577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:48.810848    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 14:04:48.813577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:48.810908    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 14:04:49.524744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:49.524709    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"
Oct 30 14:04:49.804560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:49.802754    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 14:04:49.804560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:49.802809    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 14:04:50.821563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:50.803902    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 14:04:50.821563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:50.803999    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 14:04:51.260993 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:51.260813    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:04:51.812565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:51.810087    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 14:04:51.812565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:51.810163    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 14:04:52.643065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:52.643028    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:04:53.249060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:53.249019    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:53.250647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:53.250625    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:53.251679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:53.251660    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:04:54.031878 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.031840    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"]
Oct 30 14:04:54.036933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.036912    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.055804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.055784    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-39529e-openshift-default-dockercfg-l4zq2\""
Oct 30 14:04:54.073948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.073914    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"]
Oct 30 14:04:54.147277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.147233    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.147473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.147284    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.147473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.147332    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxv58\" (UniqueName: \"kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-kube-api-access-mxv58\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.147473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.147396    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6723427-e816-42ce-96f7-841d0d6538e3-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.147473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.147412    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6723427-e816-42ce-96f7-841d0d6538e3-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.147473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.147448    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.147772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.147486    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.147772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.147508    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.147772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.147557    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.248513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.248475    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxv58\" (UniqueName: \"kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-kube-api-access-mxv58\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.248708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.248567    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6723427-e816-42ce-96f7-841d0d6538e3-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.248708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.248590    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6723427-e816-42ce-96f7-841d0d6538e3-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.248708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.248621    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.248708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.248697    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.248961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.248731    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.248961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.248772    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.248961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.248799    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.248961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.248844    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.249270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.249242    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.249403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.249338    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.249460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.249405    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6723427-e816-42ce-96f7-841d0d6538e3-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.249508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.249493    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.249692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.249662    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.251424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.251404    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.251595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.251579    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6723427-e816-42ce-96f7-841d0d6538e3-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.275654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.275625    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.275854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.275834    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxv58\" (UniqueName: \"kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-kube-api-access-mxv58\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.351411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.351367    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:54.744508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.743313    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:54.744508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.743464    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:54.744508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.743571    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:54.750046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:54.750016    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"]
Oct 30 14:04:55.743294 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:55.743247    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd" event={"ID":"e6723427-e816-42ce-96f7-841d0d6538e3","Type":"ContainerStarted","Data":"1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51"}
Oct 30 14:04:55.743696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:55.743309    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd" event={"ID":"e6723427-e816-42ce-96f7-841d0d6538e3","Type":"ContainerStarted","Data":"0310c8992f58635d2b2e0cb4f869774642003c850f5386e214363d1cb42933bc"}
Oct 30 14:04:55.830567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:55.827575    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd" podStartSLOduration=1.82755325 podStartE2EDuration="1.82755325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:55.802872436 +0000 UTC m=+960.834771376" watchObservedRunningTime="2026-10-30 14:04:55.82755325 +0000 UTC m=+960.859452181"
Oct 30 14:04:56.351598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:56.351558    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:04:56.353190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:56.353152    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 14:04:56.353300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:56.353219    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd" podUID="e6723427-e816-42ce-96f7-841d0d6538e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 14:04:57.352106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:57.352073    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 14:04:57.352641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:57.352134    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd" podUID="e6723427-e816-42ce-96f7-841d0d6538e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 14:04:57.455948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:57.455889    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:04:57.456283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:57.456256    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:04:58.352278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:58.352242    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 14:04:58.353726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:58.352293    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd" podUID="e6723427-e816-42ce-96f7-841d0d6538e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 14:04:58.774994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:58.774889    4530 generic.go:356] "Generic (PLEG): container finished" podID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerID="963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b" exitCode=0
Oct 30 14:04:58.774994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:58.774970    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" event={"ID":"9b5c1a9e-e28f-4501-ab13-a4099e4220d3","Type":"ContainerDied","Data":"963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b"}
Oct 30 14:04:58.775485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:58.775464    4530 scope.go:122] "RemoveContainer" containerID="963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b"
Oct 30 14:04:58.776976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:58.776943    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:58.777091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:58.777010    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:58.777091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:58.777064    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:04:59.352777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:59.352731    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 14:04:59.353310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:59.352798    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd" podUID="e6723427-e816-42ce-96f7-841d0d6538e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 14:04:59.731402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:59.731276    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:04:59.787612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:04:59.786686    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" event={"ID":"9b5c1a9e-e28f-4501-ab13-a4099e4220d3","Type":"ContainerStarted","Data":"78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d"}
Oct 30 14:05:00.074496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:00.074459    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:05:00.074713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:00.074606    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 14:05:00.074713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:00.074644    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 14:05:00.352524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:00.352444    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 14:05:00.352524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:00.352497    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd" podUID="e6723427-e816-42ce-96f7-841d0d6538e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 14:05:01.074446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:01.074406    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 14:05:01.074854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:01.074465    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 14:05:01.356652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:01.356573    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:05:01.357018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:01.356994    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:05:01.357770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:01.357746    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:05:02.074858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:02.074827    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 14:05:02.075221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:02.074879    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 14:05:02.824198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:02.824117    4530 generic.go:356] "Generic (PLEG): container finished" podID="9da0e269-c913-4d18-8831-b9e89e184737" containerID="420afa151307904bbc62416a306c9a02e3604e2529c4cfeba67812c50ed01035" exitCode=0
Oct 30 14:05:02.824198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:02.824187    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" event={"ID":"9da0e269-c913-4d18-8831-b9e89e184737","Type":"ContainerDied","Data":"420afa151307904bbc62416a306c9a02e3604e2529c4cfeba67812c50ed01035"}
Oct 30 14:05:02.824782 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:02.824763    4530 scope.go:122] "RemoveContainer" containerID="420afa151307904bbc62416a306c9a02e3604e2529c4cfeba67812c50ed01035"
Oct 30 14:05:02.825918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:02.825885    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:02.826017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:02.825954    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:02.826017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:02.825985    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:03.074992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:03.074916    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 14:05:03.074992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:03.074967    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 14:05:03.845518 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:03.845480    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" event={"ID":"9da0e269-c913-4d18-8831-b9e89e184737","Type":"ContainerStarted","Data":"d0b0375ede5611c6eb6773998aba1949e408b41a2d46c471f8d022b2bbafd7f2"}
Oct 30 14:05:04.074651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:04.074615    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 14:05:04.074812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:04.074670    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 14:05:04.522563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:04.521920    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:05:04.522563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:04.522050    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 14:05:04.522563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:04.522095    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 14:05:05.074986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:05.074956    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 14:05:05.075163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:05.075001    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 14:05:05.520722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:05.520399    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 14:05:05.520722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:05.520463    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 14:05:06.078285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:06.078255    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:05:06.078711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:06.078658    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:05:06.079328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:06.079306    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:05:06.519217 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:06.519136    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 14:05:06.519217 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:06.519191    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 14:05:07.519602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:07.519564    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 14:05:07.520003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:07.519619    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 14:05:08.519686 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:08.519649    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 14:05:08.520077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:08.519711    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 14:05:09.563152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:09.563114    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:05:09.564360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:09.564330    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:05:09.565011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:09.564988    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:05:10.505871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:10.505831    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:05:10.847572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:10.847526    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"]
Oct 30 14:05:10.851438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:10.851414    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:10.874757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:10.874737    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"kube-root-ca.crt\""
Oct 30 14:05:10.874935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:10.874779    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"openshift-service-ca.crt\""
Oct 30 14:05:10.877948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:10.877923    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"]
Oct 30 14:05:10.886576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:10.886556    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"default-dockercfg-wfvcl\""
Oct 30 14:05:10.887219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:10.887205    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"simulator-w0-2df0ab-kserve-self-signed-certs\""
Oct 30 14:05:11.041739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.041711    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96aa8bcc-9208-4741-8cd4-f961070a462f-tls-certs\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.041866 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.041779    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-tmp-dir\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.041866 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.041823    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw7pl\" (UniqueName: \"kubernetes.io/projected/96aa8bcc-9208-4741-8cd4-f961070a462f-kube-api-access-rw7pl\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.041943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.041903    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-dshm\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.041991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.041977    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-home\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.042033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.042021    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-model-cache\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.144046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.143422    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96aa8bcc-9208-4741-8cd4-f961070a462f-tls-certs\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.144227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.144098    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-tmp-dir\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.144227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.144139    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw7pl\" (UniqueName: \"kubernetes.io/projected/96aa8bcc-9208-4741-8cd4-f961070a462f-kube-api-access-rw7pl\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.144227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.144169    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-dshm\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.144378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.144240    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-home\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.144378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.144281    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-model-cache\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.158652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.157902    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-model-cache\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.173376 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.172986    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-home\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.178816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.178785    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-tmp-dir\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.179163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.179137    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96aa8bcc-9208-4741-8cd4-f961070a462f-tls-certs\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.182803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.182777    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-dshm\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.184855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.184833    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw7pl\" (UniqueName: \"kubernetes.io/projected/96aa8bcc-9208-4741-8cd4-f961070a462f-kube-api-access-rw7pl\") pod \"simulator-w0-2df0ab-kserve-84d768799d-btlbp\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") " pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.393197 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:05:11.393139    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:05:11.462923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.462845    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:11.763091 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:05:11.763056    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96aa8bcc_9208_4741_8cd4_f961070a462f.slice/crio-e10fd1f1bdfb96801eee158908730faf964dd7d40b311336aaa280cd698a7993 WatchSource:0}: Error finding container e10fd1f1bdfb96801eee158908730faf964dd7d40b311336aaa280cd698a7993: Status 404 returned error can't find the container with id e10fd1f1bdfb96801eee158908730faf964dd7d40b311336aaa280cd698a7993
Oct 30 14:05:11.763666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.763628    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"]
Oct 30 14:05:11.945391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.945356    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" event={"ID":"96aa8bcc-9208-4741-8cd4-f961070a462f","Type":"ContainerStarted","Data":"a754a11ca9215ab7589a6028f94a83926204b6759cbe9147bc00f3752c9e2870"}
Oct 30 14:05:11.945837 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.945403    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" event={"ID":"96aa8bcc-9208-4741-8cd4-f961070a462f","Type":"ContainerStarted","Data":"e10fd1f1bdfb96801eee158908730faf964dd7d40b311336aaa280cd698a7993"}
Oct 30 14:05:11.992111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:11.992063    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podStartSLOduration=1.992047752 podStartE2EDuration="1.992047752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:11.991596082 +0000 UTC m=+977.023495024" watchObservedRunningTime="2026-10-30 14:05:11.992047752 +0000 UTC m=+977.023946690"
Oct 30 14:05:12.463505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:12.463475    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:21.466692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:21.465847    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:21.510470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:21.510177    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:22.027132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:22.027103    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:05:22.959257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:22.959222    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:05:24.275500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.275461    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"]
Oct 30 14:05:24.280578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.279704    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.312484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.312452    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"]
Oct 30 14:05:24.367582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.367530    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-da78c4-openshift-default-dockercfg-jkr4w\""
Oct 30 14:05:24.408159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.408121    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf59q\" (UniqueName: \"kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-kube-api-access-gf59q\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.408327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.408176    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4156f6-7078-48ed-8a52-9fc04b755118-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.408327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.408254    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.408327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.408285    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.408327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.408311    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.408505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.408435    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4156f6-7078-48ed-8a52-9fc04b755118-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.408505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.408501    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.408611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.408533    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.408611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.408603    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.509714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.509677    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.509895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.509751    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf59q\" (UniqueName: \"kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-kube-api-access-gf59q\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.509895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.509789    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4156f6-7078-48ed-8a52-9fc04b755118-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.509895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.509808    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.509895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.509840    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.509895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.509868    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.510170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.509915    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4156f6-7078-48ed-8a52-9fc04b755118-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.510170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.509983    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.510170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.510040    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.510326 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.510281    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.510452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.510427    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.510661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.510639    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4156f6-7078-48ed-8a52-9fc04b755118-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.510799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.510714    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.510872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.510804    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.512460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.512438    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.512667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.512649    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4156f6-7078-48ed-8a52-9fc04b755118-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.536338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.536288    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.536630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.536608    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf59q\" (UniqueName: \"kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-kube-api-access-gf59q\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.619632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.619601    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:24.955447 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.955419    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"]
Oct 30 14:05:24.956143 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:05:24.956105    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4156f6_7078_48ed_8a52_9fc04b755118.slice/crio-5af30b6fc480fc2ad01c8c0b79cad5fcf97398fecac02369246d9d042e33f72e WatchSource:0}: Error finding container 5af30b6fc480fc2ad01c8c0b79cad5fcf97398fecac02369246d9d042e33f72e: Status 404 returned error can't find the container with id 5af30b6fc480fc2ad01c8c0b79cad5fcf97398fecac02369246d9d042e33f72e
Oct 30 14:05:24.958905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.958868    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:24.958998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.958936    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:24.958998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:24.958969    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:25.050391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:25.050359    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" event={"ID":"da4156f6-7078-48ed-8a52-9fc04b755118","Type":"ContainerStarted","Data":"5af30b6fc480fc2ad01c8c0b79cad5fcf97398fecac02369246d9d042e33f72e"}
Oct 30 14:05:26.083616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:26.083084    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" event={"ID":"da4156f6-7078-48ed-8a52-9fc04b755118","Type":"ContainerStarted","Data":"54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec"}
Oct 30 14:05:26.146237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:26.146186    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" podStartSLOduration=2.146173678 podStartE2EDuration="2.146173678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:26.143261702 +0000 UTC m=+991.175160786" watchObservedRunningTime="2026-10-30 14:05:26.146173678 +0000 UTC m=+991.178072617"
Oct 30 14:05:26.620635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:26.620600    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:26.622212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:26.622185    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 14:05:26.622330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:26.622245    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 14:05:27.620087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:27.620044    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 14:05:27.620505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:27.620113    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 14:05:28.452131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:28.451833    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:05:28.452131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:28.451902    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:05:28.639564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:28.631510    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 14:05:28.639564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:28.631583    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 14:05:29.620972 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:29.620933    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 14:05:29.621147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:29.620990    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 14:05:30.620895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:30.620853    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 14:05:30.621301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:30.620914    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 14:05:31.162268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:31.162231    4530 generic.go:356] "Generic (PLEG): container finished" podID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerID="0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873" exitCode=0
Oct 30 14:05:31.162450 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:31.162303    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" event={"ID":"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a","Type":"ContainerDied","Data":"0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873"}
Oct 30 14:05:31.162768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:31.162749    4530 scope.go:122] "RemoveContainer" containerID="0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873"
Oct 30 14:05:31.164133 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:31.164105    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:31.164192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:31.164178    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:31.164228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:31.164215    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:31.637361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:31.637327    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:31.638396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:31.638375    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:31.639138 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:31.639114    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:05:32.171194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:32.171162    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" event={"ID":"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a","Type":"ContainerStarted","Data":"f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9"}
Oct 30 14:05:32.640704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:32.640669    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:05:32.641121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:32.640837    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 14:05:32.641121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:32.640889    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 14:05:33.641150 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:33.641117    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 14:05:33.641568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:33.641173    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 14:05:34.641060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:34.641020    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 14:05:34.641238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:34.641076    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 14:05:34.818176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:34.818144    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:05:35.641025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:35.640989    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 14:05:35.641205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:35.641044    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 14:05:36.641504 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:36.641471    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 14:05:36.641914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:36.641522    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 14:05:37.644569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:37.644515    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:05:37.645031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:37.644818    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:05:37.645636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:37.645618    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:05:38.894986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:38.894949    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"]
Oct 30 14:05:38.899237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:38.899211    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:38.942028 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:38.941985    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2df0ab-openshift-default-dockercfg-z8znz\""
Oct 30 14:05:38.943759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:38.943711    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"]
Oct 30 14:05:39.047805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.047774    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.047971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.047816    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259f2601-7ea2-43b0-abeb-73891558a746-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.047971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.047934    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.048065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.047974    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.048065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.047994    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.048065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.048015    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hnk\" (UniqueName: \"kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-kube-api-access-q6hnk\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.048214 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.048175    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.048259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.048241    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259f2601-7ea2-43b0-abeb-73891558a746-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.048369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.048347    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.149936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.149861    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.149936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.149900    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259f2601-7ea2-43b0-abeb-73891558a746-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.150156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.149957    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.150156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.149995    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.150156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.150021    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.150407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.150206    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hnk\" (UniqueName: \"kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-kube-api-access-q6hnk\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.150407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.150274    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.150407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.150284    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.150407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.150359    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259f2601-7ea2-43b0-abeb-73891558a746-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.150664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.150440    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.150664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.150470    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.150664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.150504    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.150874 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.150846    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.151116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.151094    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259f2601-7ea2-43b0-abeb-73891558a746-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.152273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.152251    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.152517 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.152500    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259f2601-7ea2-43b0-abeb-73891558a746-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.177841 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.177816    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.177997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.177976    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hnk\" (UniqueName: \"kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-kube-api-access-q6hnk\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.182657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.182633    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"]
Oct 30 14:05:39.188498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.188480    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.201634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.201611    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"]
Oct 30 14:05:39.206059 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.206040    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.208573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.208553    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"]
Oct 30 14:05:39.211784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.211760    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:39.231205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.231178    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"]
Oct 30 14:05:39.352400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.352368    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.352400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.352402    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.352608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.352433    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.352608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.352535    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.352676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.352614    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.352676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.352635    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.352751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.352736    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.352784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.352762    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.352816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.352799    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.352864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.352850    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbsd\" (UniqueName: \"kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-kube-api-access-rhbsd\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.352908 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.352887    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf3a53-2882-438c-90ca-389a46799113-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.352988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.352973    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.353032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.353000    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.353085 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.353071    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.353225 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.353213    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.353268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.353256    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgq9j\" (UniqueName: \"kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-kube-api-access-pgq9j\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.353310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.353284    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.353441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.353419    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf3a53-2882-438c-90ca-389a46799113-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.454144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454073    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.454144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454106    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.454144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454124    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.454407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454162    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.454407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454190    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.454407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454215    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.454407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454250    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbsd\" (UniqueName: \"kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-kube-api-access-rhbsd\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.454407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454306    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf3a53-2882-438c-90ca-389a46799113-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.454407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454384    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.454765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454417    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.454765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454528    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.454765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454573    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.454765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454637    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.454765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454680    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgq9j\" (UniqueName: \"kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-kube-api-access-pgq9j\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.454765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454720    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.454765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454738    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.455122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454786    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.455643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.455217    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf3a53-2882-438c-90ca-389a46799113-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.455643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.454750    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf3a53-2882-438c-90ca-389a46799113-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.455643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.455268    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.455643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.455297    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.455643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.455303    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.455643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.455247    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.455643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.455338    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.455643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.455392    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.455643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.455483    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.455643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.455563    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.456076 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.455717    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.456904 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.456873    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.457941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.457915    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.458031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.457915    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.458124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.458103    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf3a53-2882-438c-90ca-389a46799113-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.481916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.481892    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.482268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.482155    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbsd\" (UniqueName: \"kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-kube-api-access-rhbsd\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.482412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.482389    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.482549 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.482520    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgq9j\" (UniqueName: \"kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-kube-api-access-pgq9j\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.512520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.512494    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:39.535379 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.535358    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:39.548967 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:05:39.548934    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259f2601_7ea2_43b0_abeb_73891558a746.slice/crio-553f11896397f6284a4ddbe31cc729070c1a6c85343f36de4c2f39220ca1c823 WatchSource:0}: Error finding container 553f11896397f6284a4ddbe31cc729070c1a6c85343f36de4c2f39220ca1c823: Status 404 returned error can't find the container with id 553f11896397f6284a4ddbe31cc729070c1a6c85343f36de4c2f39220ca1c823
Oct 30 14:05:39.550483 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.550456    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"]
Oct 30 14:05:39.551919 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.551883    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:39.551985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.551970    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:39.552030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.552014    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:39.835212 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:05:39.835170    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacf3a53_2882_438c_90ca_389a46799113.slice/crio-97c434334757df75ee633a49cb1c37da963c8905dfcc65065da539b079954f1f WatchSource:0}: Error finding container 97c434334757df75ee633a49cb1c37da963c8905dfcc65065da539b079954f1f: Status 404 returned error can't find the container with id 97c434334757df75ee633a49cb1c37da963c8905dfcc65065da539b079954f1f
Oct 30 14:05:39.836248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.836198    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"]
Oct 30 14:05:39.838117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.838084    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:39.838221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.838156    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:39.838221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.838201    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:39.855625 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:05:39.855599    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8d1f93_81d0_4f01_b25f_939c53b4b6cf.slice/crio-1c0e4a0ea792f02991bbd9966768c57745ca3a65d9ec125d1d87aafb5ce824f1 WatchSource:0}: Error finding container 1c0e4a0ea792f02991bbd9966768c57745ca3a65d9ec125d1d87aafb5ce824f1: Status 404 returned error can't find the container with id 1c0e4a0ea792f02991bbd9966768c57745ca3a65d9ec125d1d87aafb5ce824f1
Oct 30 14:05:39.856491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.856467    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"]
Oct 30 14:05:39.858964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.858932    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:39.859051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.859000    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:39.859051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:39.859042    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:40.253284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.253186    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" event={"ID":"259f2601-7ea2-43b0-abeb-73891558a746","Type":"ContainerStarted","Data":"2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6"}
Oct 30 14:05:40.253284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.253247    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" event={"ID":"259f2601-7ea2-43b0-abeb-73891558a746","Type":"ContainerStarted","Data":"553f11896397f6284a4ddbe31cc729070c1a6c85343f36de4c2f39220ca1c823"}
Oct 30 14:05:40.255142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.255112    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" event={"ID":"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf","Type":"ContainerStarted","Data":"a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5"}
Oct 30 14:05:40.255286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.255168    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" event={"ID":"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf","Type":"ContainerStarted","Data":"1c0e4a0ea792f02991bbd9966768c57745ca3a65d9ec125d1d87aafb5ce824f1"}
Oct 30 14:05:40.257017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.256992    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" event={"ID":"cacf3a53-2882-438c-90ca-389a46799113","Type":"ContainerStarted","Data":"fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801"}
Oct 30 14:05:40.257101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.257035    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" event={"ID":"cacf3a53-2882-438c-90ca-389a46799113","Type":"ContainerStarted","Data":"97c434334757df75ee633a49cb1c37da963c8905dfcc65065da539b079954f1f"}
Oct 30 14:05:40.457638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.457572    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" podStartSLOduration=1.45753234 podStartE2EDuration="1.45753234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:40.453251447 +0000 UTC m=+1005.485150387" watchObservedRunningTime="2026-10-30 14:05:40.45753234 +0000 UTC m=+1005.489431281"
Oct 30 14:05:40.458006 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.457968    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" podStartSLOduration=2.457957692 podStartE2EDuration="2.457957692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:40.394581902 +0000 UTC m=+1005.426480866" watchObservedRunningTime="2026-10-30 14:05:40.457957692 +0000 UTC m=+1005.489856630"
Oct 30 14:05:40.530619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.524577    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:40.530619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.526466    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" podStartSLOduration=1.52644673 podStartE2EDuration="1.52644673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:40.512047451 +0000 UTC m=+1005.543946391" watchObservedRunningTime="2026-10-30 14:05:40.52644673 +0000 UTC m=+1005.558345662"
Oct 30 14:05:40.530619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.530373    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 14:05:40.530619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.530424    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" podUID="cacf3a53-2882-438c-90ca-389a46799113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 14:05:40.537571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.535682    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:40.542623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.541139    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 14:05:40.542623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:40.541198    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 14:05:41.215563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:41.214952    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:41.219564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:41.218640    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 14:05:41.219564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:41.218701    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" podUID="259f2601-7ea2-43b0-abeb-73891558a746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 14:05:41.376861 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:05:41.376826    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:05:41.513625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:41.512968    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 14:05:41.513625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:41.513033    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" podUID="cacf3a53-2882-438c-90ca-389a46799113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 14:05:41.536565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:41.535949    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 14:05:41.536565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:41.536011    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 14:05:42.213566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:42.212930    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 14:05:42.213566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:42.212994    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" podUID="259f2601-7ea2-43b0-abeb-73891558a746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 14:05:42.514624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:42.514050    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 14:05:42.514624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:42.514128    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" podUID="cacf3a53-2882-438c-90ca-389a46799113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 14:05:42.540565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:42.538565    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 14:05:42.540565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:42.538621    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 14:05:43.214964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:43.214928    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 14:05:43.215211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:43.215184    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" podUID="259f2601-7ea2-43b0-abeb-73891558a746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 14:05:43.513622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:43.512933    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 14:05:43.513622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:43.512995    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" podUID="cacf3a53-2882-438c-90ca-389a46799113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 14:05:43.542565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:43.537860    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 14:05:43.542565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:43.537914    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 14:05:44.214581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:44.213896    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 14:05:44.214581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:44.213968    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" podUID="259f2601-7ea2-43b0-abeb-73891558a746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 14:05:44.519045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:44.518947    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 14:05:44.519308 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:44.519279    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" podUID="cacf3a53-2882-438c-90ca-389a46799113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 14:05:44.535922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:44.535890    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 14:05:44.536134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:44.536110    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 14:05:45.216339 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:45.216301    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 14:05:45.216967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:45.216936    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" podUID="259f2601-7ea2-43b0-abeb-73891558a746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 14:05:45.513423 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:45.512936    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 14:05:45.513423 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:45.512998    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" podUID="cacf3a53-2882-438c-90ca-389a46799113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 14:05:45.545570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:45.544805    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 14:05:45.559582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:45.552746    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 14:05:46.352405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:46.352371    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:46.355242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:46.354274    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:46.355471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:46.355449    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:05:46.736161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:46.736082    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:46.738670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:46.737348    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:46.740008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:46.739986    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:05:46.782349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:46.782312    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:46.783455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:46.783432    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:46.785028 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:46.784997    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:05:50.837282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:50.837245    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:05:54.245845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.245812    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"]
Oct 30 14:05:54.251050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.251030    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.271084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.271056    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"]
Oct 30 14:05:54.295993 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.295963    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.296122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.296006    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.296122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.296072    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.296122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.296113    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.296299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.296137    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.296299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.296152    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.296299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.296276    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skh2w\" (UniqueName: \"kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-kube-api-access-skh2w\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.296461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.296337    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.296461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.296391    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.397869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.397836    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.397993 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.397885    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skh2w\" (UniqueName: \"kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-kube-api-access-skh2w\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.397993 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.397934    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.397993 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.397962    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.397993 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.397983    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.398212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.398108    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.398212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.398153    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.398321 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.398213    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.398321 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.398253    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.398321 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.398283    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.398473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.398426    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.398569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.398529    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.398648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.398600    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.398878 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.398857    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.400252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.400230    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.400567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.400518    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.428856 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.428833    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.428954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.428924    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skh2w\" (UniqueName: \"kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-kube-api-access-skh2w\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.565202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.565171    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:54.898255 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:05:54.898186    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1650f8b_ce39_4004_84cf_c0f5dd8b0b5b.slice/crio-1ac181415b5ad756b52687659bfca77ae57c8bfd65412e0bfcd9936d4c1cd482 WatchSource:0}: Error finding container 1ac181415b5ad756b52687659bfca77ae57c8bfd65412e0bfcd9936d4c1cd482: Status 404 returned error can't find the container with id 1ac181415b5ad756b52687659bfca77ae57c8bfd65412e0bfcd9936d4c1cd482
Oct 30 14:05:54.902833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.902736    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:54.905073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.903014    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:54.905073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.903736    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:05:54.907581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:54.907394    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"]
Oct 30 14:05:55.413055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:55.413000    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" event={"ID":"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b","Type":"ContainerStarted","Data":"90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de"}
Oct 30 14:05:55.413414 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:55.413077    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" event={"ID":"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b","Type":"ContainerStarted","Data":"1ac181415b5ad756b52687659bfca77ae57c8bfd65412e0bfcd9936d4c1cd482"}
Oct 30 14:05:55.465651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:55.465607    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" podStartSLOduration=1.465592149 podStartE2EDuration="1.465592149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:55.463348074 +0000 UTC m=+1020.495247012" watchObservedRunningTime="2026-10-30 14:05:55.465592149 +0000 UTC m=+1020.497491087"
Oct 30 14:05:55.565478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:55.565446    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:05:55.567012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:55.566992    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 14:05:55.567061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:55.567034    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" podUID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 14:05:56.565880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:56.565847    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 14:05:56.566271 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:56.565897    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" podUID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 14:05:57.566126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:57.566093    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 14:05:57.566488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:57.566145    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" podUID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 14:05:58.566291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:58.566253    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 14:05:58.566688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:58.566313    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" podUID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 14:05:59.565998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:59.565957    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 14:05:59.566168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:05:59.566013    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" podUID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 14:06:00.566481 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:00.566447    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 14:06:00.566891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:00.566505    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" podUID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 14:06:01.569917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:01.569888    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:06:01.570301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:01.570269    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:06:01.570984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:01.570966    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:06:11.368715 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:06:11.368637    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:06:13.775240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:13.775204    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:06:15.657708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.657675    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"]
Oct 30 14:06:15.661439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.661421    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:15.681103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.681074    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"]
Oct 30 14:06:15.690523 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.690502    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:06:15.690653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.690599    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:06:15.802709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.802677    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n96kx\" (UniqueName: \"kubernetes.io/projected/e8618966-c007-4f7f-ac6c-70e43fb8c614-kube-api-access-n96kx\") pod \"maas-api-revoke-sub-37e8566e-7lwtl\" (UID: \"e8618966-c007-4f7f-ac6c-70e43fb8c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:15.802863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.802830    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37e8566e-7lwtl\" (UID: \"e8618966-c007-4f7f-ac6c-70e43fb8c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:15.802958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.802945    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37e8566e-7lwtl\" (UID: \"e8618966-c007-4f7f-ac6c-70e43fb8c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:15.908616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.906443    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37e8566e-7lwtl\" (UID: \"e8618966-c007-4f7f-ac6c-70e43fb8c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:15.908616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.906502    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n96kx\" (UniqueName: \"kubernetes.io/projected/e8618966-c007-4f7f-ac6c-70e43fb8c614-kube-api-access-n96kx\") pod \"maas-api-revoke-sub-37e8566e-7lwtl\" (UID: \"e8618966-c007-4f7f-ac6c-70e43fb8c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:15.908616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.906663    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37e8566e-7lwtl\" (UID: \"e8618966-c007-4f7f-ac6c-70e43fb8c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:15.908616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.907991    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37e8566e-7lwtl\" (UID: \"e8618966-c007-4f7f-ac6c-70e43fb8c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:15.915915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.915884    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37e8566e-7lwtl\" (UID: \"e8618966-c007-4f7f-ac6c-70e43fb8c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:15.939968 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.939937    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n96kx\" (UniqueName: \"kubernetes.io/projected/e8618966-c007-4f7f-ac6c-70e43fb8c614-kube-api-access-n96kx\") pod \"maas-api-revoke-sub-37e8566e-7lwtl\" (UID: \"e8618966-c007-4f7f-ac6c-70e43fb8c614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:15.974912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:15.974886    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:16.276717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:16.275961    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"]
Oct 30 14:06:16.278726 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:06:16.278665    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8618966_c007_4f7f_ac6c_70e43fb8c614.slice/crio-ec8246774422def25ac85d416964029364b38af73e82cd5e8dd333133fbc838d WatchSource:0}: Error finding container ec8246774422def25ac85d416964029364b38af73e82cd5e8dd333133fbc838d: Status 404 returned error can't find the container with id ec8246774422def25ac85d416964029364b38af73e82cd5e8dd333133fbc838d
Oct 30 14:06:16.280341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:16.280323    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:06:16.597976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:16.597944    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl" event={"ID":"e8618966-c007-4f7f-ac6c-70e43fb8c614","Type":"ContainerStarted","Data":"ec8246774422def25ac85d416964029364b38af73e82cd5e8dd333133fbc838d"}
Oct 30 14:06:19.625367 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:19.625278    4530 generic.go:356] "Generic (PLEG): container finished" podID="e8618966-c007-4f7f-ac6c-70e43fb8c614" containerID="b2396cde600d5639280aac21873c254654a44d4ef280133168b557bca31bd06b" exitCode=0
Oct 30 14:06:19.625749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:19.625368    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl" event={"ID":"e8618966-c007-4f7f-ac6c-70e43fb8c614","Type":"ContainerDied","Data":"b2396cde600d5639280aac21873c254654a44d4ef280133168b557bca31bd06b"}
Oct 30 14:06:20.824381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:20.824356    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:20.964849 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:20.964779    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-xks-ca\") pod \"e8618966-c007-4f7f-ac6c-70e43fb8c614\" (UID: \"e8618966-c007-4f7f-ac6c-70e43fb8c614\") "
Oct 30 14:06:20.964976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:20.964880    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8618966-c007-4f7f-ac6c-70e43fb8c614-kube-api-access-n96kx\" (UniqueName: \"kubernetes.io/projected/e8618966-c007-4f7f-ac6c-70e43fb8c614-kube-api-access-n96kx\") pod \"e8618966-c007-4f7f-ac6c-70e43fb8c614\" (UID: \"e8618966-c007-4f7f-ac6c-70e43fb8c614\") "
Oct 30 14:06:20.964976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:20.964953    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-openshift-ca\") pod \"e8618966-c007-4f7f-ac6c-70e43fb8c614\" (UID: \"e8618966-c007-4f7f-ac6c-70e43fb8c614\") "
Oct 30 14:06:20.965259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:20.965236    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-openshift-ca" pod "e8618966-c007-4f7f-ac6c-70e43fb8c614" (UID: "e8618966-c007-4f7f-ac6c-70e43fb8c614"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:20.966902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:20.966879    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-xks-ca" pod "e8618966-c007-4f7f-ac6c-70e43fb8c614" (UID: "e8618966-c007-4f7f-ac6c-70e43fb8c614"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:06:20.967015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:20.967000    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8618966-c007-4f7f-ac6c-70e43fb8c614-kube-api-access-n96kx" pod "e8618966-c007-4f7f-ac6c-70e43fb8c614" (UID: "e8618966-c007-4f7f-ac6c-70e43fb8c614"). InnerVolumeSpecName "kube-api-access-n96kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:21.065996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:21.065969    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n96kx\" (UniqueName: \"kubernetes.io/projected/e8618966-c007-4f7f-ac6c-70e43fb8c614-kube-api-access-n96kx\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:06:21.065996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:21.065992    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:06:21.066146 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:21.066002    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8618966-c007-4f7f-ac6c-70e43fb8c614-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:06:21.674325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:21.674279    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl" event={"ID":"e8618966-c007-4f7f-ac6c-70e43fb8c614","Type":"ContainerDied","Data":"ec8246774422def25ac85d416964029364b38af73e82cd5e8dd333133fbc838d"}
Oct 30 14:06:21.674495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:21.674350    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec8246774422def25ac85d416964029364b38af73e82cd5e8dd333133fbc838d"
Oct 30 14:06:21.674495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:21.674300    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"
Oct 30 14:06:24.050342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.050310    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"]
Oct 30 14:06:24.050847 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.050833    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e8618966-c007-4f7f-ac6c-70e43fb8c614" containerName="revoke-keys"
Oct 30 14:06:24.050890 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.050848    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8618966-c007-4f7f-ac6c-70e43fb8c614" containerName="revoke-keys"
Oct 30 14:06:24.050955 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.050946    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8618966-c007-4f7f-ac6c-70e43fb8c614" containerName="revoke-keys"
Oct 30 14:06:24.054517 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.054499    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.083252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.083227    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"]
Oct 30 14:06:24.219633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.219599    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.219809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.219638    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.219809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.219662    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kr5t\" (UniqueName: \"kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-kube-api-access-8kr5t\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.219809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.219791    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.219975 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.219841    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.219975 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.219950    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.220081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.219997    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.220081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.220047    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.220235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.220211    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.321719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.321652    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.321719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.321695    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.321719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.321711    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.321977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.321732    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kr5t\" (UniqueName: \"kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-kube-api-access-8kr5t\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.321977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.321854    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.321977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.321888    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.321977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.321960    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.322180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.321990    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.322180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.322011    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.322180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.322122    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.322284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.322186    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.322284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.322223    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.322368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.322347    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.322699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.322679    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.324364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.324337    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.324487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.324470    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.348341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.348319    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.348666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.348646    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kr5t\" (UniqueName: \"kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-kube-api-access-8kr5t\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.369131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.369111    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:24.475492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.475460    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"]
Oct 30 14:06:24.480679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.480655    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.506729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.506700    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"]
Oct 30 14:06:24.625479 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.625402    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.625479 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.625449    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.625690 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.625602    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.625690 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.625641    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.625806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.625758    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.625885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.625804    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fztdf\" (UniqueName: \"kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-kube-api-access-fztdf\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.625885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.625839    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.625958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.625921    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.626052 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.626027    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.685877 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:06:24.685843    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3753a7f2_b5bb_465d_85c1_3e2f86b57cfa.slice/crio-8bc90fd620d47086765900a8e8fe51555a6b0f49ec4cb4563fa4949db0de12d2 WatchSource:0}: Error finding container 8bc90fd620d47086765900a8e8fe51555a6b0f49ec4cb4563fa4949db0de12d2: Status 404 returned error can't find the container with id 8bc90fd620d47086765900a8e8fe51555a6b0f49ec4cb4563fa4949db0de12d2
Oct 30 14:06:24.687016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.686998    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"]
Oct 30 14:06:24.688255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.688226    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:06:24.688306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.688287    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:06:24.688344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.688314    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:06:24.727390 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.727367    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.727488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.727413    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.727488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.727478    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.727653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.727512    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztdf\" (UniqueName: \"kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-kube-api-access-fztdf\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.727653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.727583    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.727653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.727618    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.727811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.727701    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.727811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.727749    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.727811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.727786    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.727944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.727834    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.728130 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.728101    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.728240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.728136    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.728667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.728642    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.728732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.728662    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.729765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.729744    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.730220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.730203    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.754088 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.754061    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.754189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.754141    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztdf\" (UniqueName: \"kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-kube-api-access-fztdf\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:24.799259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:24.799224    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:25.120004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:25.119973    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"]
Oct 30 14:06:25.120338 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:06:25.120120    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef3270b_eec9_4ca5_9b4d_d6faf51ce1d7.slice/crio-f27b940ff062a0b7bc82794e69bbae2353ac6a97a077a8b7c6de01311a059cb2 WatchSource:0}: Error finding container f27b940ff062a0b7bc82794e69bbae2353ac6a97a077a8b7c6de01311a059cb2: Status 404 returned error can't find the container with id f27b940ff062a0b7bc82794e69bbae2353ac6a97a077a8b7c6de01311a059cb2
Oct 30 14:06:25.123781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:25.123448    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:06:25.123781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:25.123532    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:06:25.123781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:25.123614    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:06:25.746350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:25.746302    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" event={"ID":"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7","Type":"ContainerStarted","Data":"74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184"}
Oct 30 14:06:25.746573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:25.746409    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" event={"ID":"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7","Type":"ContainerStarted","Data":"f27b940ff062a0b7bc82794e69bbae2353ac6a97a077a8b7c6de01311a059cb2"}
Oct 30 14:06:25.748662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:25.748636    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" event={"ID":"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa","Type":"ContainerStarted","Data":"05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b"}
Oct 30 14:06:25.748773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:25.748689    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" event={"ID":"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa","Type":"ContainerStarted","Data":"8bc90fd620d47086765900a8e8fe51555a6b0f49ec4cb4563fa4949db0de12d2"}
Oct 30 14:06:25.800672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:25.800630    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:25.838647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:25.838567    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" podStartSLOduration=1.8385236470000001 podStartE2EDuration="1.838523647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:25.818485971 +0000 UTC m=+1050.850384910" watchObservedRunningTime="2026-10-30 14:06:25.838523647 +0000 UTC m=+1050.870422588"
Oct 30 14:06:25.872952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:25.872905    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" podStartSLOduration=1.8728926160000001 podStartE2EDuration="1.872892616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:25.86936617 +0000 UTC m=+1050.901265108" watchObservedRunningTime="2026-10-30 14:06:25.872892616 +0000 UTC m=+1050.904791603"
Oct 30 14:06:26.371224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:26.370199    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:26.371658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:26.371286    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 14:06:26.371658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:26.371364    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" podUID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 14:06:26.807566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:26.806703    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 14:06:26.807566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:26.806762    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 14:06:27.369815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:27.369781    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 14:06:27.369986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:27.369843    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" podUID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 14:06:27.808561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:27.799873    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 14:06:27.808561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:27.799934    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 14:06:28.369411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:28.369379    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 14:06:28.369721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:28.369436    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" podUID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 14:06:28.804559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:28.801871    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 14:06:28.804559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:28.801929    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 14:06:29.370411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:29.370376    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 14:06:29.370813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:29.370429    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" podUID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 14:06:29.832011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:29.826112    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 14:06:29.832011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:29.826181    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 14:06:30.155801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.155709    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"]
Oct 30 14:06:30.162344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.162315    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:30.208566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.176330    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"]
Oct 30 14:06:30.208566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.195584    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:06:30.208566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.203668    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:06:30.239404 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.239366    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-557aac5b-xw9k4\" (UID: \"147baac4-8f45-4a17-9728-1d3483a300f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:30.239598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.239471    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-557aac5b-xw9k4\" (UID: \"147baac4-8f45-4a17-9728-1d3483a300f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:30.239598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.239514    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh5p7\" (UniqueName: \"kubernetes.io/projected/147baac4-8f45-4a17-9728-1d3483a300f3-kube-api-access-rh5p7\") pod \"maas-api-revoke-sub-557aac5b-xw9k4\" (UID: \"147baac4-8f45-4a17-9728-1d3483a300f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:30.340383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.340346    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh5p7\" (UniqueName: \"kubernetes.io/projected/147baac4-8f45-4a17-9728-1d3483a300f3-kube-api-access-rh5p7\") pod \"maas-api-revoke-sub-557aac5b-xw9k4\" (UID: \"147baac4-8f45-4a17-9728-1d3483a300f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:30.340582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.340564    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-557aac5b-xw9k4\" (UID: \"147baac4-8f45-4a17-9728-1d3483a300f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:30.340654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.340607    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-557aac5b-xw9k4\" (UID: \"147baac4-8f45-4a17-9728-1d3483a300f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:30.341328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.341304    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-557aac5b-xw9k4\" (UID: \"147baac4-8f45-4a17-9728-1d3483a300f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:30.342936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.342912    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-557aac5b-xw9k4\" (UID: \"147baac4-8f45-4a17-9728-1d3483a300f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:30.367178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.367162    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh5p7\" (UniqueName: \"kubernetes.io/projected/147baac4-8f45-4a17-9728-1d3483a300f3-kube-api-access-rh5p7\") pod \"maas-api-revoke-sub-557aac5b-xw9k4\" (UID: \"147baac4-8f45-4a17-9728-1d3483a300f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:30.370081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.370060    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 14:06:30.370157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.370101    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" podUID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 14:06:30.496685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.483737    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:30.835014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.834957    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 14:06:30.835208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.835028    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 14:06:30.868245 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:06:30.868220    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147baac4_8f45_4a17_9728_1d3483a300f3.slice/crio-81c99911718adf35f624833c775bf8efe178368aeff9f875b57152d96cf69450 WatchSource:0}: Error finding container 81c99911718adf35f624833c775bf8efe178368aeff9f875b57152d96cf69450: Status 404 returned error can't find the container with id 81c99911718adf35f624833c775bf8efe178368aeff9f875b57152d96cf69450
Oct 30 14:06:30.869338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:30.869317    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"]
Oct 30 14:06:31.631378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:31.631343    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:31.631998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:31.631973    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:31.633392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:31.633361    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:06:31.800006 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:31.799975    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 14:06:31.800192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:31.800023    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 14:06:31.858879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:31.853896    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4" event={"ID":"147baac4-8f45-4a17-9728-1d3483a300f3","Type":"ContainerStarted","Data":"8da7b5505d26d471cac4bad856e3dd0b35da769b99bc13b606dfc642b8edbe0c"}
Oct 30 14:06:31.858879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:31.853957    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4" event={"ID":"147baac4-8f45-4a17-9728-1d3483a300f3","Type":"ContainerStarted","Data":"81c99911718adf35f624833c775bf8efe178368aeff9f875b57152d96cf69450"}
Oct 30 14:06:31.903108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:31.903059    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4" podStartSLOduration=1.9030454460000001 podStartE2EDuration="1.903045446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:31.899505281 +0000 UTC m=+1056.931404221" watchObservedRunningTime="2026-10-30 14:06:31.903045446 +0000 UTC m=+1056.934944385"
Oct 30 14:06:32.804432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:32.804396    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:32.804849 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:32.804731    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:32.805438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:32.805420    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:06:32.859764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:32.859733    4530 generic.go:356] "Generic (PLEG): container finished" podID="147baac4-8f45-4a17-9728-1d3483a300f3" containerID="8da7b5505d26d471cac4bad856e3dd0b35da769b99bc13b606dfc642b8edbe0c" exitCode=0
Oct 30 14:06:32.859914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:32.859808    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4" event={"ID":"147baac4-8f45-4a17-9728-1d3483a300f3","Type":"ContainerDied","Data":"8da7b5505d26d471cac4bad856e3dd0b35da769b99bc13b606dfc642b8edbe0c"}
Oct 30 14:06:33.998887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:33.998861    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:34.097319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.097273    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-openshift-ca\") pod \"147baac4-8f45-4a17-9728-1d3483a300f3\" (UID: \"147baac4-8f45-4a17-9728-1d3483a300f3\") "
Oct 30 14:06:34.097501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.097446    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147baac4-8f45-4a17-9728-1d3483a300f3-kube-api-access-rh5p7\" (UniqueName: \"kubernetes.io/projected/147baac4-8f45-4a17-9728-1d3483a300f3-kube-api-access-rh5p7\") pod \"147baac4-8f45-4a17-9728-1d3483a300f3\" (UID: \"147baac4-8f45-4a17-9728-1d3483a300f3\") "
Oct 30 14:06:34.097501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.097491    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-xks-ca\") pod \"147baac4-8f45-4a17-9728-1d3483a300f3\" (UID: \"147baac4-8f45-4a17-9728-1d3483a300f3\") "
Oct 30 14:06:34.097709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.097685    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-openshift-ca" pod "147baac4-8f45-4a17-9728-1d3483a300f3" (UID: "147baac4-8f45-4a17-9728-1d3483a300f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:34.097850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.097833    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:06:34.099484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.099460    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-xks-ca" pod "147baac4-8f45-4a17-9728-1d3483a300f3" (UID: "147baac4-8f45-4a17-9728-1d3483a300f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:06:34.099561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.099490    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147baac4-8f45-4a17-9728-1d3483a300f3-kube-api-access-rh5p7" pod "147baac4-8f45-4a17-9728-1d3483a300f3" (UID: "147baac4-8f45-4a17-9728-1d3483a300f3"). InnerVolumeSpecName "kube-api-access-rh5p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:34.198472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.198416    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh5p7\" (UniqueName: \"kubernetes.io/projected/147baac4-8f45-4a17-9728-1d3483a300f3-kube-api-access-rh5p7\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:06:34.198472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.198439    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147baac4-8f45-4a17-9728-1d3483a300f3-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:06:34.822308 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.822271    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:06:34.874175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.874141    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4" event={"ID":"147baac4-8f45-4a17-9728-1d3483a300f3","Type":"ContainerDied","Data":"81c99911718adf35f624833c775bf8efe178368aeff9f875b57152d96cf69450"}
Oct 30 14:06:34.874363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.874195    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81c99911718adf35f624833c775bf8efe178368aeff9f875b57152d96cf69450"
Oct 30 14:06:34.874363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:34.874155    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"
Oct 30 14:06:41.367360 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:06:41.367322    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:06:43.941786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:43.941754    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"]
Oct 30 14:06:43.942329 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:43.942314    4530 cpu_manager.go:414] "containerMap: removing container" podUID="147baac4-8f45-4a17-9728-1d3483a300f3" containerName="revoke-keys"
Oct 30 14:06:43.942383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:43.942331    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147baac4-8f45-4a17-9728-1d3483a300f3" containerName="revoke-keys"
Oct 30 14:06:43.942441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:43.942430    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="147baac4-8f45-4a17-9728-1d3483a300f3" containerName="revoke-keys"
Oct 30 14:06:43.945690 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:43.945672    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:43.968280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:43.968253    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"]
Oct 30 14:06:43.978418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:43.978392    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:06:43.978579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:43.978462    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:06:44.136983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:44.136953    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c576cb31-fjhds\" (UID: \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:44.137138 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:44.137103    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmk6x\" (UniqueName: \"kubernetes.io/projected/0dd19176-0f16-4091-a38f-4b51aaaa80c2-kube-api-access-fmk6x\") pod \"maas-api-revoke-sub-c576cb31-fjhds\" (UID: \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:44.137180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:44.137167    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c576cb31-fjhds\" (UID: \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:44.237790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:44.237713    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c576cb31-fjhds\" (UID: \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:44.237942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:44.237861    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmk6x\" (UniqueName: \"kubernetes.io/projected/0dd19176-0f16-4091-a38f-4b51aaaa80c2-kube-api-access-fmk6x\") pod \"maas-api-revoke-sub-c576cb31-fjhds\" (UID: \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:44.238009 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:44.237952    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c576cb31-fjhds\" (UID: \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:44.238312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:44.238290    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c576cb31-fjhds\" (UID: \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:44.240371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:44.240349    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c576cb31-fjhds\" (UID: \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:44.265263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:44.265239    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmk6x\" (UniqueName: \"kubernetes.io/projected/0dd19176-0f16-4091-a38f-4b51aaaa80c2-kube-api-access-fmk6x\") pod \"maas-api-revoke-sub-c576cb31-fjhds\" (UID: \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:44.558719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:44.558694    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:44.861852 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:06:44.861820    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd19176_0f16_4091_a38f_4b51aaaa80c2.slice/crio-aec9572dee11818900f92c89cd89748460682bf8d6572d486dff3d1bab9dfaf8 WatchSource:0}: Error finding container aec9572dee11818900f92c89cd89748460682bf8d6572d486dff3d1bab9dfaf8: Status 404 returned error can't find the container with id aec9572dee11818900f92c89cd89748460682bf8d6572d486dff3d1bab9dfaf8
Oct 30 14:06:44.863053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:44.863030    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"]
Oct 30 14:06:44.983598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:44.983566    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds" event={"ID":"0dd19176-0f16-4091-a38f-4b51aaaa80c2","Type":"ContainerStarted","Data":"aec9572dee11818900f92c89cd89748460682bf8d6572d486dff3d1bab9dfaf8"}
Oct 30 14:06:45.991065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:45.990984    4530 generic.go:356] "Generic (PLEG): container finished" podID="0dd19176-0f16-4091-a38f-4b51aaaa80c2" containerID="9fe678cf72af0778416bd42b2770809e6402db1b4947243229a334bab94892a2" exitCode=0
Oct 30 14:06:45.991426 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:45.991066    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds" event={"ID":"0dd19176-0f16-4091-a38f-4b51aaaa80c2","Type":"ContainerDied","Data":"9fe678cf72af0778416bd42b2770809e6402db1b4947243229a334bab94892a2"}
Oct 30 14:06:47.132923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:47.132897    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:47.173568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:47.173368    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd19176-0f16-4091-a38f-4b51aaaa80c2-kube-api-access-fmk6x\" (UniqueName: \"kubernetes.io/projected/0dd19176-0f16-4091-a38f-4b51aaaa80c2-kube-api-access-fmk6x\") pod \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\" (UID: \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\") "
Oct 30 14:06:47.173568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:47.173485    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-xks-ca\") pod \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\" (UID: \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\") "
Oct 30 14:06:47.173568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:47.173556    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-openshift-ca\") pod \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\" (UID: \"0dd19176-0f16-4091-a38f-4b51aaaa80c2\") "
Oct 30 14:06:47.212178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:47.207365    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-openshift-ca" pod "0dd19176-0f16-4091-a38f-4b51aaaa80c2" (UID: "0dd19176-0f16-4091-a38f-4b51aaaa80c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:47.212178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:47.211086    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-xks-ca" pod "0dd19176-0f16-4091-a38f-4b51aaaa80c2" (UID: "0dd19176-0f16-4091-a38f-4b51aaaa80c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:06:47.212178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:47.211220    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd19176-0f16-4091-a38f-4b51aaaa80c2-kube-api-access-fmk6x" pod "0dd19176-0f16-4091-a38f-4b51aaaa80c2" (UID: "0dd19176-0f16-4091-a38f-4b51aaaa80c2"). InnerVolumeSpecName "kube-api-access-fmk6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:47.274692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:47.274611    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmk6x\" (UniqueName: \"kubernetes.io/projected/0dd19176-0f16-4091-a38f-4b51aaaa80c2-kube-api-access-fmk6x\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:06:47.274692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:47.274642    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:06:47.274692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:47.274658    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd19176-0f16-4091-a38f-4b51aaaa80c2-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:06:48.005358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:48.005328    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"
Oct 30 14:06:48.005517 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:48.005373    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds" event={"ID":"0dd19176-0f16-4091-a38f-4b51aaaa80c2","Type":"ContainerDied","Data":"aec9572dee11818900f92c89cd89748460682bf8d6572d486dff3d1bab9dfaf8"}
Oct 30 14:06:48.005517 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:48.005431    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aec9572dee11818900f92c89cd89748460682bf8d6572d486dff3d1bab9dfaf8"
Oct 30 14:06:54.263613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.263580    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"]
Oct 30 14:06:54.264145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.264130    4530 cpu_manager.go:414] "containerMap: removing container" podUID="0dd19176-0f16-4091-a38f-4b51aaaa80c2" containerName="revoke-keys"
Oct 30 14:06:54.264188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.264146    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd19176-0f16-4091-a38f-4b51aaaa80c2" containerName="revoke-keys"
Oct 30 14:06:54.264239 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.264230    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd19176-0f16-4091-a38f-4b51aaaa80c2" containerName="revoke-keys"
Oct 30 14:06:54.271137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.271116    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:54.288110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.288083    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"]
Oct 30 14:06:54.297150 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.297127    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:06:54.297420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.297401    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:06:54.369375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.369340    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5365de31-dk558\" (UID: \"02bf1ede-db4e-4233-a082-ed75c386272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:54.369555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.369494    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5365de31-dk558\" (UID: \"02bf1ede-db4e-4233-a082-ed75c386272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:54.369647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.369626    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsh59\" (UniqueName: \"kubernetes.io/projected/02bf1ede-db4e-4233-a082-ed75c386272c-kube-api-access-bsh59\") pod \"maas-api-revoke-sub-5365de31-dk558\" (UID: \"02bf1ede-db4e-4233-a082-ed75c386272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:54.471411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.471377    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5365de31-dk558\" (UID: \"02bf1ede-db4e-4233-a082-ed75c386272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:54.471537 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.471468    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5365de31-dk558\" (UID: \"02bf1ede-db4e-4233-a082-ed75c386272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:54.471616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.471531    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsh59\" (UniqueName: \"kubernetes.io/projected/02bf1ede-db4e-4233-a082-ed75c386272c-kube-api-access-bsh59\") pod \"maas-api-revoke-sub-5365de31-dk558\" (UID: \"02bf1ede-db4e-4233-a082-ed75c386272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:54.472204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.472181    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5365de31-dk558\" (UID: \"02bf1ede-db4e-4233-a082-ed75c386272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:54.473863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.473842    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5365de31-dk558\" (UID: \"02bf1ede-db4e-4233-a082-ed75c386272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:54.498017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.497992    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsh59\" (UniqueName: \"kubernetes.io/projected/02bf1ede-db4e-4233-a082-ed75c386272c-kube-api-access-bsh59\") pod \"maas-api-revoke-sub-5365de31-dk558\" (UID: \"02bf1ede-db4e-4233-a082-ed75c386272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:54.586094 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.586073    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:54.908613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:54.905749    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"]
Oct 30 14:06:54.927061 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:06:54.927014    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bf1ede_db4e_4233_a082_ed75c386272c.slice/crio-d998073af94fee2d7323ea3820a007bec01754da5d2e4dafcee63a17ad339388 WatchSource:0}: Error finding container d998073af94fee2d7323ea3820a007bec01754da5d2e4dafcee63a17ad339388: Status 404 returned error can't find the container with id d998073af94fee2d7323ea3820a007bec01754da5d2e4dafcee63a17ad339388
Oct 30 14:06:55.118453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:55.118423    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558" event={"ID":"02bf1ede-db4e-4233-a082-ed75c386272c","Type":"ContainerStarted","Data":"d998073af94fee2d7323ea3820a007bec01754da5d2e4dafcee63a17ad339388"}
Oct 30 14:06:56.126902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:56.126865    4530 generic.go:356] "Generic (PLEG): container finished" podID="02bf1ede-db4e-4233-a082-ed75c386272c" containerID="6d206a67a2983c402aedbd17f5a006da1db1c8af57475aaddab0da8c8bae3040" exitCode=0
Oct 30 14:06:56.127274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:56.126950    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558" event={"ID":"02bf1ede-db4e-4233-a082-ed75c386272c","Type":"ContainerDied","Data":"6d206a67a2983c402aedbd17f5a006da1db1c8af57475aaddab0da8c8bae3040"}
Oct 30 14:06:57.285303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:57.285277    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:57.411670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:57.411595    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bf1ede-db4e-4233-a082-ed75c386272c-kube-api-access-bsh59\" (UniqueName: \"kubernetes.io/projected/02bf1ede-db4e-4233-a082-ed75c386272c-kube-api-access-bsh59\") pod \"02bf1ede-db4e-4233-a082-ed75c386272c\" (UID: \"02bf1ede-db4e-4233-a082-ed75c386272c\") "
Oct 30 14:06:57.411670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:57.411661    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-xks-ca\") pod \"02bf1ede-db4e-4233-a082-ed75c386272c\" (UID: \"02bf1ede-db4e-4233-a082-ed75c386272c\") "
Oct 30 14:06:57.411852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:57.411748    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-openshift-ca\") pod \"02bf1ede-db4e-4233-a082-ed75c386272c\" (UID: \"02bf1ede-db4e-4233-a082-ed75c386272c\") "
Oct 30 14:06:57.412131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:57.412099    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-openshift-ca" pod "02bf1ede-db4e-4233-a082-ed75c386272c" (UID: "02bf1ede-db4e-4233-a082-ed75c386272c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:57.413696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:57.413671    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-xks-ca" pod "02bf1ede-db4e-4233-a082-ed75c386272c" (UID: "02bf1ede-db4e-4233-a082-ed75c386272c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:06:57.413696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:57.413671    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bf1ede-db4e-4233-a082-ed75c386272c-kube-api-access-bsh59" pod "02bf1ede-db4e-4233-a082-ed75c386272c" (UID: "02bf1ede-db4e-4233-a082-ed75c386272c"). InnerVolumeSpecName "kube-api-access-bsh59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:57.512976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:57.512948    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:06:57.512976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:57.512972    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsh59\" (UniqueName: \"kubernetes.io/projected/02bf1ede-db4e-4233-a082-ed75c386272c-kube-api-access-bsh59\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:06:57.513104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:57.512982    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02bf1ede-db4e-4233-a082-ed75c386272c-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:06:58.141329 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:58.141298    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"
Oct 30 14:06:58.141495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:58.141345    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558" event={"ID":"02bf1ede-db4e-4233-a082-ed75c386272c","Type":"ContainerDied","Data":"d998073af94fee2d7323ea3820a007bec01754da5d2e4dafcee63a17ad339388"}
Oct 30 14:06:58.141495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:06:58.141403    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d998073af94fee2d7323ea3820a007bec01754da5d2e4dafcee63a17ad339388"
Oct 30 14:07:00.368525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:00.368494    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:07:02.636412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.636374    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"]
Oct 30 14:07:02.637895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.637866    4530 cpu_manager.go:414] "containerMap: removing container" podUID="02bf1ede-db4e-4233-a082-ed75c386272c" containerName="revoke-keys"
Oct 30 14:07:02.638055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.638039    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bf1ede-db4e-4233-a082-ed75c386272c" containerName="revoke-keys"
Oct 30 14:07:02.638292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.638273    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bf1ede-db4e-4233-a082-ed75c386272c" containerName="revoke-keys"
Oct 30 14:07:02.651137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.651109    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:02.651283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.651183    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"]
Oct 30 14:07:02.669456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.669422    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:07:02.709372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.709334    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a12cdd11-q4zd8\" (UID: \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:02.709756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.709725    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zzpf\" (UniqueName: \"kubernetes.io/projected/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-kube-api-access-9zzpf\") pod \"maas-api-revoke-sub-a12cdd11-q4zd8\" (UID: \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:02.709973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.709952    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a12cdd11-q4zd8\" (UID: \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:02.710474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.710453    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:07:02.811067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.811019    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zzpf\" (UniqueName: \"kubernetes.io/projected/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-kube-api-access-9zzpf\") pod \"maas-api-revoke-sub-a12cdd11-q4zd8\" (UID: \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:02.811399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.811375    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a12cdd11-q4zd8\" (UID: \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:02.812359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.812229    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a12cdd11-q4zd8\" (UID: \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:02.812530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.812502    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a12cdd11-q4zd8\" (UID: \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:02.828273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.828252    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a12cdd11-q4zd8\" (UID: \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:02.849039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.849011    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zzpf\" (UniqueName: \"kubernetes.io/projected/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-kube-api-access-9zzpf\") pod \"maas-api-revoke-sub-a12cdd11-q4zd8\" (UID: \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:02.969525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:02.969457    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:03.175841 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.175736    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"]
Oct 30 14:07:03.186511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.182837    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:03.254623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.240528    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d22b72fe-jvwhl\" (UID: \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:03.254623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.240602    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98c6w\" (UniqueName: \"kubernetes.io/projected/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-kube-api-access-98c6w\") pod \"maas-api-revoke-sub-d22b72fe-jvwhl\" (UID: \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:03.254623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.240669    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d22b72fe-jvwhl\" (UID: \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:03.259652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.259615    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"]
Oct 30 14:07:03.343564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.341631    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d22b72fe-jvwhl\" (UID: \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:03.343564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.341688    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98c6w\" (UniqueName: \"kubernetes.io/projected/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-kube-api-access-98c6w\") pod \"maas-api-revoke-sub-d22b72fe-jvwhl\" (UID: \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:03.343564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.341752    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d22b72fe-jvwhl\" (UID: \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:03.346086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.345060    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d22b72fe-jvwhl\" (UID: \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:03.346086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.345791    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d22b72fe-jvwhl\" (UID: \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:03.372424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.372399    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98c6w\" (UniqueName: \"kubernetes.io/projected/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-kube-api-access-98c6w\") pod \"maas-api-revoke-sub-d22b72fe-jvwhl\" (UID: \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:03.434183 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:07:03.434145    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a2052e_e5b5_4a35_b33b_d094d2d9f800.slice/crio-e630021cdcb5942ba21bd7d6507d1291e6b6561a3492b2e68b868e96151a4cbb WatchSource:0}: Error finding container e630021cdcb5942ba21bd7d6507d1291e6b6561a3492b2e68b868e96151a4cbb: Status 404 returned error can't find the container with id e630021cdcb5942ba21bd7d6507d1291e6b6561a3492b2e68b868e96151a4cbb
Oct 30 14:07:03.438364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.438318    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"]
Oct 30 14:07:03.574905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.574875    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:03.920697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:03.912904    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"]
Oct 30 14:07:04.192435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:04.192350    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl" event={"ID":"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8","Type":"ContainerStarted","Data":"b0d61e9b61b495ddad8a152236ae0acb5db5781682da2eba3fe6dbcc9ddfda57"}
Oct 30 14:07:04.194224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:04.194182    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8" event={"ID":"e4a2052e-e5b5-4a35-b33b-d094d2d9f800","Type":"ContainerStarted","Data":"7344c0d086647a220b860108e1d489c16976ddfddd4472e259a2f696467a09e7"}
Oct 30 14:07:04.194370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:04.194229    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8" event={"ID":"e4a2052e-e5b5-4a35-b33b-d094d2d9f800","Type":"ContainerStarted","Data":"e630021cdcb5942ba21bd7d6507d1291e6b6561a3492b2e68b868e96151a4cbb"}
Oct 30 14:07:04.242226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:04.242181    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8" podStartSLOduration=2.242166405 podStartE2EDuration="2.242166405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:04.239059489 +0000 UTC m=+1089.270958430" watchObservedRunningTime="2026-10-30 14:07:04.242166405 +0000 UTC m=+1089.274065343"
Oct 30 14:07:05.209369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:05.209331    4530 generic.go:356] "Generic (PLEG): container finished" podID="8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8" containerID="86acecf15adbf4bc2b571582ea2c7118785f9e6d4a9c927e6634427ead70f450" exitCode=0
Oct 30 14:07:05.209780 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:05.209390    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl" event={"ID":"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8","Type":"ContainerDied","Data":"86acecf15adbf4bc2b571582ea2c7118785f9e6d4a9c927e6634427ead70f450"}
Oct 30 14:07:05.211189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:05.211166    4530 generic.go:356] "Generic (PLEG): container finished" podID="e4a2052e-e5b5-4a35-b33b-d094d2d9f800" containerID="7344c0d086647a220b860108e1d489c16976ddfddd4472e259a2f696467a09e7" exitCode=0
Oct 30 14:07:05.211295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:05.211196    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8" event={"ID":"e4a2052e-e5b5-4a35-b33b-d094d2d9f800","Type":"ContainerDied","Data":"7344c0d086647a220b860108e1d489c16976ddfddd4472e259a2f696467a09e7"}
Oct 30 14:07:06.397916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.397888    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:06.401702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.401683    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:06.488603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.488524    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-xks-ca\") pod \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\" (UID: \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\") "
Oct 30 14:07:06.488603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.488585    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-openshift-ca\") pod \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\" (UID: \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\") "
Oct 30 14:07:06.488815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.488733    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-kube-api-access-9zzpf\" (UniqueName: \"kubernetes.io/projected/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-kube-api-access-9zzpf\") pod \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\" (UID: \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\") "
Oct 30 14:07:06.488815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.488767    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-openshift-ca\") pod \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\" (UID: \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\") "
Oct 30 14:07:06.488815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.488797    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-kube-api-access-98c6w\" (UniqueName: \"kubernetes.io/projected/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-kube-api-access-98c6w\") pod \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\" (UID: \"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8\") "
Oct 30 14:07:06.488998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.488837    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-xks-ca\") pod \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\" (UID: \"e4a2052e-e5b5-4a35-b33b-d094d2d9f800\") "
Oct 30 14:07:06.488998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.488946    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-openshift-ca" pod "e4a2052e-e5b5-4a35-b33b-d094d2d9f800" (UID: "e4a2052e-e5b5-4a35-b33b-d094d2d9f800"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:06.489420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.489268    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-openshift-ca" pod "8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8" (UID: "8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:06.489420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.489387    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:07:06.489420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.489400    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:07:06.491505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.491476    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-kube-api-access-9zzpf" pod "e4a2052e-e5b5-4a35-b33b-d094d2d9f800" (UID: "e4a2052e-e5b5-4a35-b33b-d094d2d9f800"). InnerVolumeSpecName "kube-api-access-9zzpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:06.491667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.491607    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-xks-ca" pod "e4a2052e-e5b5-4a35-b33b-d094d2d9f800" (UID: "e4a2052e-e5b5-4a35-b33b-d094d2d9f800"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:06.492082 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.492059    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-xks-ca" pod "8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8" (UID: "8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:06.492626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.492518    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-kube-api-access-98c6w" pod "8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8" (UID: "8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8"). InnerVolumeSpecName "kube-api-access-98c6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:06.590242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.590213    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zzpf\" (UniqueName: \"kubernetes.io/projected/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-kube-api-access-9zzpf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:07:06.590242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.590240    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98c6w\" (UniqueName: \"kubernetes.io/projected/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-kube-api-access-98c6w\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:07:06.590385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.590249    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a2052e-e5b5-4a35-b33b-d094d2d9f800-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:07:06.590385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:06.590258    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:07:07.225985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:07.225962    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"
Oct 30 14:07:07.225985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:07.225964    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl" event={"ID":"8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8","Type":"ContainerDied","Data":"b0d61e9b61b495ddad8a152236ae0acb5db5781682da2eba3fe6dbcc9ddfda57"}
Oct 30 14:07:07.226193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:07.226019    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0d61e9b61b495ddad8a152236ae0acb5db5781682da2eba3fe6dbcc9ddfda57"
Oct 30 14:07:07.227515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:07.227492    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8" event={"ID":"e4a2052e-e5b5-4a35-b33b-d094d2d9f800","Type":"ContainerDied","Data":"e630021cdcb5942ba21bd7d6507d1291e6b6561a3492b2e68b868e96151a4cbb"}
Oct 30 14:07:07.227652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:07.227530    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e630021cdcb5942ba21bd7d6507d1291e6b6561a3492b2e68b868e96151a4cbb"
Oct 30 14:07:07.227652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:07.227500    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"
Oct 30 14:07:10.676867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.676830    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"]
Oct 30 14:07:10.677374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.677357    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8" containerName="revoke-keys"
Oct 30 14:07:10.677443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.677375    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8" containerName="revoke-keys"
Oct 30 14:07:10.677443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.677406    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e4a2052e-e5b5-4a35-b33b-d094d2d9f800" containerName="revoke-keys"
Oct 30 14:07:10.677443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.677411    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a2052e-e5b5-4a35-b33b-d094d2d9f800" containerName="revoke-keys"
Oct 30 14:07:10.677628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.677501    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a2052e-e5b5-4a35-b33b-d094d2d9f800" containerName="revoke-keys"
Oct 30 14:07:10.677628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.677512    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8" containerName="revoke-keys"
Oct 30 14:07:10.682011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.681988    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:10.700122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.700099    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"]
Oct 30 14:07:10.709774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.709754    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:07:10.709901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.709852    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:07:10.752360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.752328    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r68fx\" (UniqueName: \"kubernetes.io/projected/e3a5e790-56ad-4ad5-b69d-a65c874f7903-kube-api-access-r68fx\") pod \"maas-api-revoke-sub-99469b37-vqs7r\" (UID: \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:10.752486 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.752467    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99469b37-vqs7r\" (UID: \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:10.752534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.752492    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99469b37-vqs7r\" (UID: \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:10.853424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.853387    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99469b37-vqs7r\" (UID: \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:10.853424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.853423    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99469b37-vqs7r\" (UID: \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:10.853680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.853556    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68fx\" (UniqueName: \"kubernetes.io/projected/e3a5e790-56ad-4ad5-b69d-a65c874f7903-kube-api-access-r68fx\") pod \"maas-api-revoke-sub-99469b37-vqs7r\" (UID: \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:10.854193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.854165    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99469b37-vqs7r\" (UID: \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:10.855862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.855833    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99469b37-vqs7r\" (UID: \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:10.879895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.879874    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68fx\" (UniqueName: \"kubernetes.io/projected/e3a5e790-56ad-4ad5-b69d-a65c874f7903-kube-api-access-r68fx\") pod \"maas-api-revoke-sub-99469b37-vqs7r\" (UID: \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:10.998346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:10.998293    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:11.302112 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:07:11.302085    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a5e790_56ad_4ad5_b69d_a65c874f7903.slice/crio-8f53e7b6d05fc75e0c787232b15c526bb4619a379be0593494c023170fb5d812 WatchSource:0}: Error finding container 8f53e7b6d05fc75e0c787232b15c526bb4619a379be0593494c023170fb5d812: Status 404 returned error can't find the container with id 8f53e7b6d05fc75e0c787232b15c526bb4619a379be0593494c023170fb5d812
Oct 30 14:07:11.304134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:11.304110    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"]
Oct 30 14:07:11.365571 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:07:11.365530    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:07:12.262222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:12.262139    4530 generic.go:356] "Generic (PLEG): container finished" podID="e3a5e790-56ad-4ad5-b69d-a65c874f7903" containerID="d0be799e94af7b70224c7b46d9baca182c3925ef14e181e2ac24e9153381c561" exitCode=0
Oct 30 14:07:12.262592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:12.262226    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r" event={"ID":"e3a5e790-56ad-4ad5-b69d-a65c874f7903","Type":"ContainerDied","Data":"d0be799e94af7b70224c7b46d9baca182c3925ef14e181e2ac24e9153381c561"}
Oct 30 14:07:12.262592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:12.262283    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r" event={"ID":"e3a5e790-56ad-4ad5-b69d-a65c874f7903","Type":"ContainerStarted","Data":"8f53e7b6d05fc75e0c787232b15c526bb4619a379be0593494c023170fb5d812"}
Oct 30 14:07:13.406226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:13.406202    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:13.483239 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:13.483212    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-xks-ca\") pod \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\" (UID: \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\") "
Oct 30 14:07:13.483396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:13.483255    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-openshift-ca\") pod \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\" (UID: \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\") "
Oct 30 14:07:13.483396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:13.483288    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3a5e790-56ad-4ad5-b69d-a65c874f7903-kube-api-access-r68fx\" (UniqueName: \"kubernetes.io/projected/e3a5e790-56ad-4ad5-b69d-a65c874f7903-kube-api-access-r68fx\") pod \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\" (UID: \"e3a5e790-56ad-4ad5-b69d-a65c874f7903\") "
Oct 30 14:07:13.483779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:13.483746    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-openshift-ca" pod "e3a5e790-56ad-4ad5-b69d-a65c874f7903" (UID: "e3a5e790-56ad-4ad5-b69d-a65c874f7903"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:13.485488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:13.485458    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-xks-ca" pod "e3a5e790-56ad-4ad5-b69d-a65c874f7903" (UID: "e3a5e790-56ad-4ad5-b69d-a65c874f7903"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:13.485920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:13.485898    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a5e790-56ad-4ad5-b69d-a65c874f7903-kube-api-access-r68fx" pod "e3a5e790-56ad-4ad5-b69d-a65c874f7903" (UID: "e3a5e790-56ad-4ad5-b69d-a65c874f7903"). InnerVolumeSpecName "kube-api-access-r68fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:13.585024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:13.584993    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:07:13.585024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:13.585025    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a5e790-56ad-4ad5-b69d-a65c874f7903-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:07:13.585190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:13.585038    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r68fx\" (UniqueName: \"kubernetes.io/projected/e3a5e790-56ad-4ad5-b69d-a65c874f7903-kube-api-access-r68fx\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:07:14.276833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:14.276804    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"
Oct 30 14:07:14.277023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:14.276840    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r" event={"ID":"e3a5e790-56ad-4ad5-b69d-a65c874f7903","Type":"ContainerDied","Data":"8f53e7b6d05fc75e0c787232b15c526bb4619a379be0593494c023170fb5d812"}
Oct 30 14:07:14.277023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:14.276890    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f53e7b6d05fc75e0c787232b15c526bb4619a379be0593494c023170fb5d812"
Oct 30 14:07:17.845761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:17.845715    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:07:17.846395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:17.845798    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:07:18.093567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.074735    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"]
Oct 30 14:07:18.093567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.075562    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e3a5e790-56ad-4ad5-b69d-a65c874f7903" containerName="revoke-keys"
Oct 30 14:07:18.093567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.075584    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a5e790-56ad-4ad5-b69d-a65c874f7903" containerName="revoke-keys"
Oct 30 14:07:18.093567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.075774    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a5e790-56ad-4ad5-b69d-a65c874f7903" containerName="revoke-keys"
Oct 30 14:07:18.094277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.094247    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.190068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.182765    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.190068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.182835    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256b8107-4ad4-46c2-88f2-26c2c15fe866-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.190068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.182900    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.190068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.182975    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.190068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.183031    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.190068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.183069    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.190068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.183206    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.190068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.183235    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srfj7\" (UniqueName: \"kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-kube-api-access-srfj7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.190068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.183274    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.234439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.233338    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"]
Oct 30 14:07:18.285971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.285846    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.286902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.286877    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256b8107-4ad4-46c2-88f2-26c2c15fe866-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.288086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.288066    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.291706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.291675    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.292380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.291528    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.292697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.287957    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256b8107-4ad4-46c2-88f2-26c2c15fe866-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.292697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.286803    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.293208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.293173    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.293412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.292348    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.293694 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.293675    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.294392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.294371    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.294868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.294849    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfj7\" (UniqueName: \"kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-kube-api-access-srfj7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.295590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.295568    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.296327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.296009    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.333833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.331283    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srfj7\" (UniqueName: \"kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-kube-api-access-srfj7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.344650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.344594    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.344650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.318228    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.345589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.345523    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tfw9l\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.581707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.581675    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:18.899281 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:07:18.899248    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256b8107_4ad4_46c2_88f2_26c2c15fe866.slice/crio-4ad05b5b75742ba96305c15f4be0d01e2d4369b0226c64b81c81890507dfd89d WatchSource:0}: Error finding container 4ad05b5b75742ba96305c15f4be0d01e2d4369b0226c64b81c81890507dfd89d: Status 404 returned error can't find the container with id 4ad05b5b75742ba96305c15f4be0d01e2d4369b0226c64b81c81890507dfd89d
Oct 30 14:07:18.902131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.902091    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:18.902226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.902163    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:18.902226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.902195    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:18.904004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:18.903979    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"]
Oct 30 14:07:19.055796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.055766    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"]
Oct 30 14:07:19.059877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.059854    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:19.078245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.078224    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:07:19.078424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.078275    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"]
Oct 30 14:07:19.078596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.078436    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:07:19.207573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.207458    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glghs\" (UniqueName: \"kubernetes.io/projected/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-kube-api-access-glghs\") pod \"maas-api-revoke-sub-f774f453-dsl6k\" (UID: \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:19.207726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.207668    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f774f453-dsl6k\" (UID: \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:19.207870 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.207851    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f774f453-dsl6k\" (UID: \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:19.308986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.308953    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f774f453-dsl6k\" (UID: \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:19.309173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.309057    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f774f453-dsl6k\" (UID: \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:19.309173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.309114    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glghs\" (UniqueName: \"kubernetes.io/projected/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-kube-api-access-glghs\") pod \"maas-api-revoke-sub-f774f453-dsl6k\" (UID: \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:19.309901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.309868    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f774f453-dsl6k\" (UID: \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:19.311501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.311476    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f774f453-dsl6k\" (UID: \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:19.337812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.337788    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glghs\" (UniqueName: \"kubernetes.io/projected/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-kube-api-access-glghs\") pod \"maas-api-revoke-sub-f774f453-dsl6k\" (UID: \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:19.355604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.355567    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" event={"ID":"256b8107-4ad4-46c2-88f2-26c2c15fe866","Type":"ContainerStarted","Data":"31849c9b60d69f30b8fc64cdef65c7105bdb1235cc58a6d8b2f8f7e1323d8d66"}
Oct 30 14:07:19.355725 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.355625    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" event={"ID":"256b8107-4ad4-46c2-88f2-26c2c15fe866","Type":"ContainerStarted","Data":"4ad05b5b75742ba96305c15f4be0d01e2d4369b0226c64b81c81890507dfd89d"}
Oct 30 14:07:19.373673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.373650    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:19.432400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.432345    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" podStartSLOduration=1.4323263019999999 podStartE2EDuration="1.432326302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:19.428016539 +0000 UTC m=+1104.459915493" watchObservedRunningTime="2026-10-30 14:07:19.432326302 +0000 UTC m=+1104.464225241"
Oct 30 14:07:19.585992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.585951    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:19.587335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.587301    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 14:07:19.587482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.587374    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 14:07:19.676360 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:07:19.676314    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cd24bc_8ca3_4d0f_b7fd_5e3794a9619e.slice/crio-5349f8f75675943626441280d2824a234b51f72ee6752026b239a7947ec36b47 WatchSource:0}: Error finding container 5349f8f75675943626441280d2824a234b51f72ee6752026b239a7947ec36b47: Status 404 returned error can't find the container with id 5349f8f75675943626441280d2824a234b51f72ee6752026b239a7947ec36b47
Oct 30 14:07:19.677553 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:19.677515    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"]
Oct 30 14:07:20.363843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:20.363808    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k" event={"ID":"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e","Type":"ContainerStarted","Data":"d8b69b905690637ee40e567b3a757393b2cd4c88c4d49f2bd1945d35f64e555a"}
Oct 30 14:07:20.364298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:20.363865    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k" event={"ID":"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e","Type":"ContainerStarted","Data":"5349f8f75675943626441280d2824a234b51f72ee6752026b239a7947ec36b47"}
Oct 30 14:07:20.418219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:20.418162    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k" podStartSLOduration=1.418147012 podStartE2EDuration="1.418147012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:20.416876018 +0000 UTC m=+1105.448774956" watchObservedRunningTime="2026-10-30 14:07:20.418147012 +0000 UTC m=+1105.450045951"
Oct 30 14:07:20.604783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:20.604731    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 14:07:20.604992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:20.604812    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 14:07:21.406298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:21.406267    4530 generic.go:356] "Generic (PLEG): container finished" podID="b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e" containerID="d8b69b905690637ee40e567b3a757393b2cd4c88c4d49f2bd1945d35f64e555a" exitCode=0
Oct 30 14:07:21.407580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:21.407516    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k" event={"ID":"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e","Type":"ContainerDied","Data":"d8b69b905690637ee40e567b3a757393b2cd4c88c4d49f2bd1945d35f64e555a"}
Oct 30 14:07:21.600979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:21.600941    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 14:07:21.601264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:21.601238    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 14:07:22.604595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:22.595919    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 14:07:22.604595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:22.595977    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 14:07:22.610213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:22.610190    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:22.675990 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:22.675916    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-kube-api-access-glghs\" (UniqueName: \"kubernetes.io/projected/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-kube-api-access-glghs\") pod \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\" (UID: \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\") "
Oct 30 14:07:22.675990 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:22.675964    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-xks-ca\") pod \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\" (UID: \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\") "
Oct 30 14:07:22.676196 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:22.675999    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-openshift-ca\") pod \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\" (UID: \"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e\") "
Oct 30 14:07:22.676498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:22.676460    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-openshift-ca" pod "b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e" (UID: "b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:22.676689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:22.676662    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:07:22.678395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:22.678362    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-kube-api-access-glghs" pod "b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e" (UID: "b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e"). InnerVolumeSpecName "kube-api-access-glghs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:22.678602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:22.678536    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-xks-ca" pod "b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e" (UID: "b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:22.777338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:22.777310    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glghs\" (UniqueName: \"kubernetes.io/projected/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-kube-api-access-glghs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:07:22.777338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:22.777336    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:07:23.422825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:23.422716    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"
Oct 30 14:07:23.422825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:23.422746    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k" event={"ID":"b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e","Type":"ContainerDied","Data":"5349f8f75675943626441280d2824a234b51f72ee6752026b239a7947ec36b47"}
Oct 30 14:07:23.422825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:23.422808    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5349f8f75675943626441280d2824a234b51f72ee6752026b239a7947ec36b47"
Oct 30 14:07:23.604567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:23.583858    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 14:07:23.604567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:23.583917    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 14:07:24.404794 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.404753    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"]
Oct 30 14:07:24.405406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.405385    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e" containerName="revoke-keys"
Oct 30 14:07:24.405406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.405406    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e" containerName="revoke-keys"
Oct 30 14:07:24.405606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.405511    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e" containerName="revoke-keys"
Oct 30 14:07:24.412697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.412675    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"]
Oct 30 14:07:24.412802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.412765    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.519625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.519591    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.519625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.519623    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.519826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.519640    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.519826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.519663    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.519826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.519798    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.519948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.519927    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.519988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.519953    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm7ml\" (UniqueName: \"kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-kube-api-access-hm7ml\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.519988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.519981    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.520122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.520079    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.600914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.600869    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 14:07:24.601165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.601137    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 14:07:24.621419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.621389    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.621896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.621455    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.621896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.621508    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm7ml\" (UniqueName: \"kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-kube-api-access-hm7ml\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.621896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.621568    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.621896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.621622    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.621896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.621692    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.621896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.621736    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.621896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.621761    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.621896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.621793    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.622332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.622084    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.622332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.622114    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.622332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.622183    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.622332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.622230    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.622604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.622582    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.624015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.623991    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.624139 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.624124    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.648066 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.648032    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.648875 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.648848    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm7ml\" (UniqueName: \"kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-kube-api-access-hm7ml\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:24.729279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:24.729189    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:25.076957 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:07:25.076913    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc093100_4dd1_4d93_ae2f_9bd51f35bb5c.slice/crio-b66e25ce2429b32f6801029230429ac75ab6bc559d5cd2a8642b8aaa458f963e WatchSource:0}: Error finding container b66e25ce2429b32f6801029230429ac75ab6bc559d5cd2a8642b8aaa458f963e: Status 404 returned error can't find the container with id b66e25ce2429b32f6801029230429ac75ab6bc559d5cd2a8642b8aaa458f963e
Oct 30 14:07:25.078610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:25.078529    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"]
Oct 30 14:07:25.079397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:25.079364    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:25.079458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:25.079427    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:25.079493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:25.079458    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:25.441525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:25.441440    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" event={"ID":"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c","Type":"ContainerStarted","Data":"6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29"}
Oct 30 14:07:25.441525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:25.441508    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" event={"ID":"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c","Type":"ContainerStarted","Data":"b66e25ce2429b32f6801029230429ac75ab6bc559d5cd2a8642b8aaa458f963e"}
Oct 30 14:07:25.493668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:25.493610    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" podStartSLOduration=1.493591222 podStartE2EDuration="1.493591222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:25.493019109 +0000 UTC m=+1110.524918057" watchObservedRunningTime="2026-10-30 14:07:25.493591222 +0000 UTC m=+1110.525490162"
Oct 30 14:07:25.592565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:25.583999    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 14:07:25.592565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:25.584098    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 14:07:25.729464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:25.729371    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:26.587567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:26.583930    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 14:07:26.587567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:26.583986    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 14:07:26.731458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:26.731425    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 14:07:26.731881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:26.731477    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 14:07:27.730037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:27.730003    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 14:07:27.730216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:27.730056    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 14:07:27.857281 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:27.857243    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:27.858457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:27.858431    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:27.859777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:27.859751    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:07:28.729575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:28.729519    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 14:07:28.729772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:28.729603    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 14:07:29.729564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:29.729510    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 14:07:29.730010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:29.729623    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 14:07:30.007934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:30.007839    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:07:30.732954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:30.732917    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 14:07:30.733503 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:30.733475    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 14:07:31.729995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:31.729956    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 14:07:31.730300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:31.730269    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 14:07:32.685704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:32.685661    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:07:32.686331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:32.686283    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:07:33.222891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.222858    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"]
Oct 30 14:07:33.270853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.270820    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"]
Oct 30 14:07:33.271651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.271407    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.394668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.394633    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.394859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.394831    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.394962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.394940    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61375fe0-b572-459f-991d-169aa49b28ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.395013 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.394997    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.395158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.395135    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61375fe0-b572-459f-991d-169aa49b28ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.395235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.395188    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.395282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.395229    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmjsc\" (UniqueName: \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-kube-api-access-dmjsc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.395374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.395354    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.395453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.395390    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496047    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496106    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61375fe0-b572-459f-991d-169aa49b28ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496131    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496155    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmjsc\" (UniqueName: \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-kube-api-access-dmjsc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496213    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496238    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496269    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496369    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496424    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61375fe0-b572-459f-991d-169aa49b28ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496524    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496527    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496668    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.496909 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.496781    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.497230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.497208    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61375fe0-b572-459f-991d-169aa49b28ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.498678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.498651    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.498815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.498796    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61375fe0-b572-459f-991d-169aa49b28ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.523250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.523219    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.523335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.523315    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmjsc\" (UniqueName: \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-kube-api-access-dmjsc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ww87s\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.642065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.642035    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:33.792035 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.791949    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:07:33.792035 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.791999    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:07:33.991720 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:07:33.991680    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61375fe0_b572_459f_991d_169aa49b28ef.slice/crio-770098f2ad5f210dbbaab5164d9a2a960cedcbe17989e8b97a7b7776af614b62 WatchSource:0}: Error finding container 770098f2ad5f210dbbaab5164d9a2a960cedcbe17989e8b97a7b7776af614b62: Status 404 returned error can't find the container with id 770098f2ad5f210dbbaab5164d9a2a960cedcbe17989e8b97a7b7776af614b62
Oct 30 14:07:33.992367 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.992341    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"]
Oct 30 14:07:33.994735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.994638    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:33.994735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.994701    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:33.994910 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:33.994743    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:34.091519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:34.091484    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:34.093050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:34.093006    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:34.093922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:34.093898    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:07:34.607512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:34.607470    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" event={"ID":"61375fe0-b572-459f-991d-169aa49b28ef","Type":"ContainerStarted","Data":"7ed38d1c95d5fb3d090045b15b155365cf4d5ca08588d3d64f9135516b5aa02f"}
Oct 30 14:07:34.607717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:34.607532    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" event={"ID":"61375fe0-b572-459f-991d-169aa49b28ef","Type":"ContainerStarted","Data":"770098f2ad5f210dbbaab5164d9a2a960cedcbe17989e8b97a7b7776af614b62"}
Oct 30 14:07:34.642495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:34.642471    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:34.658771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:34.658734    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" podStartSLOduration=1.658722505 podStartE2EDuration="1.658722505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:34.657592522 +0000 UTC m=+1119.689491461" watchObservedRunningTime="2026-10-30 14:07:34.658722505 +0000 UTC m=+1119.690621443"
Oct 30 14:07:35.644327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:35.644293    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ww87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 14:07:35.644755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:35.644345    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" podUID="61375fe0-b572-459f-991d-169aa49b28ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 14:07:36.642785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:36.642746    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ww87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 14:07:36.642962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:36.642817    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" podUID="61375fe0-b572-459f-991d-169aa49b28ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 14:07:37.642612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:37.642581    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ww87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 14:07:37.642987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:37.642631    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" podUID="61375fe0-b572-459f-991d-169aa49b28ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 14:07:38.643340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:38.643307    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ww87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 14:07:38.643812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:38.643366    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" podUID="61375fe0-b572-459f-991d-169aa49b28ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 14:07:39.642748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:39.642715    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ww87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 14:07:39.642932 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:39.642778    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" podUID="61375fe0-b572-459f-991d-169aa49b28ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 14:07:40.034014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.033934    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"]
Oct 30 14:07:40.039262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.039238    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.068498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.068469    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"]
Oct 30 14:07:40.211910 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.211869    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.212072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.212001    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.212072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.212043    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.212072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.212069    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.212240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.212225    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.212277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.212256    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e166404-62ee-4c62-83b7-ba6375714a5b-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.212312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.212283    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.212347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.212318    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.212405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.212393    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4b68\" (UniqueName: \"kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-kube-api-access-r4b68\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.314176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.314135    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.314351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.314195    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e166404-62ee-4c62-83b7-ba6375714a5b-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.314351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.314240    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.314351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.314265    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.314351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.314312    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4b68\" (UniqueName: \"kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-kube-api-access-r4b68\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.314565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.314360    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.314565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.314456    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.314565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.314502    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.314739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.314584    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.315038 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.315003    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.315179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.315019    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.315179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.315056    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.315179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.315087    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e166404-62ee-4c62-83b7-ba6375714a5b-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.315303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.315203    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.316740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.316711    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.316934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.316915    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.341188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.341150    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.341327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.341308    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4b68\" (UniqueName: \"kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-kube-api-access-r4b68\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.358703 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.358681    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:40.642994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.642921    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ww87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 14:07:40.642994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.642983    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" podUID="61375fe0-b572-459f-991d-169aa49b28ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 14:07:40.681918 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:07:40.681890    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e166404_62ee_4c62_83b7_ba6375714a5b.slice/crio-9d6b38638f98a6e4d4a3040229617bba0dc3ae984b922f599d3362748b49e8b2 WatchSource:0}: Error finding container 9d6b38638f98a6e4d4a3040229617bba0dc3ae984b922f599d3362748b49e8b2: Status 404 returned error can't find the container with id 9d6b38638f98a6e4d4a3040229617bba0dc3ae984b922f599d3362748b49e8b2
Oct 30 14:07:40.685117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.685080    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:40.685213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.685150    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"]
Oct 30 14:07:40.685254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.685152    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:40.685288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:40.685264    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:41.367208 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:07:41.367172    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:07:41.643554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:41.643450    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ww87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 14:07:41.643914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:41.643525    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" podUID="61375fe0-b572-459f-991d-169aa49b28ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 14:07:41.657848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:41.657817    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" event={"ID":"0e166404-62ee-4c62-83b7-ba6375714a5b","Type":"ContainerStarted","Data":"f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338"}
Oct 30 14:07:41.657997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:41.657884    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" event={"ID":"0e166404-62ee-4c62-83b7-ba6375714a5b","Type":"ContainerStarted","Data":"9d6b38638f98a6e4d4a3040229617bba0dc3ae984b922f599d3362748b49e8b2"}
Oct 30 14:07:41.733358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:41.733300    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" podStartSLOduration=1.7332853990000001 podStartE2EDuration="1.733285399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:41.730983577 +0000 UTC m=+1126.762882531" watchObservedRunningTime="2026-10-30 14:07:41.733285399 +0000 UTC m=+1126.765184337"
Oct 30 14:07:42.359715 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:42.359675    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:42.361209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:42.361180    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 14:07:42.361337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:42.361229    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" podUID="0e166404-62ee-4c62-83b7-ba6375714a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 14:07:43.120914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:43.120875    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:43.121363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:43.121345    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:43.122593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:43.122569    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:07:43.359294 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:43.359247    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 14:07:43.359481 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:43.359308    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" podUID="0e166404-62ee-4c62-83b7-ba6375714a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 14:07:44.359307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:44.359272    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 14:07:44.359778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:44.359339    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" podUID="0e166404-62ee-4c62-83b7-ba6375714a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 14:07:45.360565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:45.359260    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 14:07:45.360565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:45.359340    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" podUID="0e166404-62ee-4c62-83b7-ba6375714a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 14:07:46.358976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:46.358940    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 14:07:46.359175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:46.359009    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" podUID="0e166404-62ee-4c62-83b7-ba6375714a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 14:07:47.491628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:47.491596    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:47.492193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:47.492081    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:47.493265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:47.493242    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:07:53.370675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:53.370643    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:07:54.808498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.808459    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"]
Oct 30 14:07:54.859804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.858792    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"]
Oct 30 14:07:54.859804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.858895    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:54.885096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.885065    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"]
Oct 30 14:07:54.889040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.889020    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:54.914964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.914937    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"]
Oct 30 14:07:54.984136 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984105    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:54.984288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984218    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849e6253-e63b-4f69-a345-15cef381ae4c-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:54.984288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984250    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:54.984361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984313    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25qxj\" (UniqueName: \"kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-kube-api-access-25qxj\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:54.984361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984346    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:54.984424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984361    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:54.984424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984377    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:54.984424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984396    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:54.984511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984447    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rgjt\" (UniqueName: \"kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-kube-api-access-7rgjt\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:54.984511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984490    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:54.984597 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984582    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:54.984642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984630    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:54.984723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984703    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849e6253-e63b-4f69-a345-15cef381ae4c-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:54.984763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984746    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:54.984808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984764    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:54.984808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984786    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:54.984930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984866    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:54.984930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:54.984883    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.086434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086352    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.086434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086405    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849e6253-e63b-4f69-a345-15cef381ae4c-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.086682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086449    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.086682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086478    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.086682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086505    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.086682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086567    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.086682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086594    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.086682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086641    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.086988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086736    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849e6253-e63b-4f69-a345-15cef381ae4c-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.086988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086766    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.086988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086810    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25qxj\" (UniqueName: \"kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-kube-api-access-25qxj\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.086988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086853    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.086988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086878    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.086988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086901    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.086988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086930    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.086988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086954    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rgjt\" (UniqueName: \"kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-kube-api-access-7rgjt\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.086988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.086982    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.087429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.087048    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.087429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.087084    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.087429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.087138    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.087429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.087170    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.087429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.087314    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.087429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.087325    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.087429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.087345    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.087855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.087646    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.087855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.087717    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.088142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.088096    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849e6253-e63b-4f69-a345-15cef381ae4c-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.088142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.088132    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.089534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.089515    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.089660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.089640    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.089986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.089968    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.090239 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.090219    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849e6253-e63b-4f69-a345-15cef381ae4c-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.115241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.115211    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.116345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.116318    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25qxj\" (UniqueName: \"kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-kube-api-access-25qxj\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.116509 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.116491    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.116586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.116518    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rgjt\" (UniqueName: \"kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-kube-api-access-7rgjt\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.177695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.177673    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:55.201708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.201686    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:55.497337 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:07:55.497312    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849e6253_e63b_4f69_a345_15cef381ae4c.slice/crio-1a3ea61cba12dc7dcb5b506ed8c1cb6a103ae04782ce8393c80aeee09154e611 WatchSource:0}: Error finding container 1a3ea61cba12dc7dcb5b506ed8c1cb6a103ae04782ce8393c80aeee09154e611: Status 404 returned error can't find the container with id 1a3ea61cba12dc7dcb5b506ed8c1cb6a103ae04782ce8393c80aeee09154e611
Oct 30 14:07:55.498828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.498801    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"]
Oct 30 14:07:55.499713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.499680    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:55.499784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.499743    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:55.499823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.499790    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:55.543583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.543555    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"]
Oct 30 14:07:55.544251 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:07:55.544223    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af3ba0a_bb15_4ae7_8542_bbf4eb7674f5.slice/crio-c8b6388da989d70205a76a03efb7ebb4f09da926ee3021ea9ae3fff5e83231ff WatchSource:0}: Error finding container c8b6388da989d70205a76a03efb7ebb4f09da926ee3021ea9ae3fff5e83231ff: Status 404 returned error can't find the container with id c8b6388da989d70205a76a03efb7ebb4f09da926ee3021ea9ae3fff5e83231ff
Oct 30 14:07:55.546750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.546712    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:55.546855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.546794    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:55.546930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.546870    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:07:55.817231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.816914    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" event={"ID":"849e6253-e63b-4f69-a345-15cef381ae4c","Type":"ContainerStarted","Data":"e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e"}
Oct 30 14:07:55.817231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.816977    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" event={"ID":"849e6253-e63b-4f69-a345-15cef381ae4c","Type":"ContainerStarted","Data":"1a3ea61cba12dc7dcb5b506ed8c1cb6a103ae04782ce8393c80aeee09154e611"}
Oct 30 14:07:55.819522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.819494    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" event={"ID":"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5","Type":"ContainerStarted","Data":"733a5ee819a4e390e3e3798928e38561fafcdeb32d6aa1e4b24c8bdb56300ced"}
Oct 30 14:07:55.819705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.819571    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" event={"ID":"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5","Type":"ContainerStarted","Data":"c8b6388da989d70205a76a03efb7ebb4f09da926ee3021ea9ae3fff5e83231ff"}
Oct 30 14:07:55.870495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.870445    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" podStartSLOduration=1.87043235 podStartE2EDuration="1.87043235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:55.868148106 +0000 UTC m=+1140.900047058" watchObservedRunningTime="2026-10-30 14:07:55.87043235 +0000 UTC m=+1140.902331288"
Oct 30 14:07:55.919235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:55.919185    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" podStartSLOduration=1.91917079 podStartE2EDuration="1.91917079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:55.918132013 +0000 UTC m=+1140.950030965" watchObservedRunningTime="2026-10-30 14:07:55.91917079 +0000 UTC m=+1140.951069729"
Oct 30 14:07:56.178217 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:56.178148    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:07:56.179338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:56.179317    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 14:07:56.179441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:56.179368    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 14:07:56.202299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:56.202263    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:07:56.203436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:56.203416    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 14:07:56.203515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:56.203457    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 14:07:57.178700 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:57.178671    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 14:07:57.179078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:57.178718    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 14:07:57.207709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:57.207675    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 14:07:57.207974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:57.207942    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 14:07:58.178349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:58.178312    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 14:07:58.178518 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:58.178368    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 14:07:58.204566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:58.203886    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 14:07:58.204566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:58.203942    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 14:07:59.178512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:59.178473    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 14:07:59.178727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:59.178523    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 14:07:59.208455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:59.208421    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 14:07:59.209024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:07:59.208997    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 14:08:00.178728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:00.178693    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 14:08:00.178914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:00.178748    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 14:08:00.210395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:00.210361    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 14:08:00.210966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:00.210935    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 14:08:01.178868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:01.178689    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 14:08:01.178868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:01.178762    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 14:08:01.223565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:01.218940    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 14:08:01.223565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:01.219007    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 14:08:02.178212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:02.178171    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 14:08:02.178412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:02.178227    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 14:08:02.202696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:02.202661    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 14:08:02.202830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:02.202719    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 14:08:03.182914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:03.182880    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:08:03.183288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:03.183158    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:08:03.183961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:03.183941    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:08:03.206552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:03.206505    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:08:03.206826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:03.206805    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:08:03.207507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:03.207490    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:08:09.360455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.360416    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"]
Oct 30 14:08:09.419556 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.419504    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"]
Oct 30 14:08:09.426564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.419858    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.552564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.523196    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.552946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.552907    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.553121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.553102    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.553261 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.553244    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.553388 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.553373    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssmdc\" (UniqueName: \"kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-kube-api-access-ssmdc\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.553589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.553570    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.553755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.553736    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.553918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.553901    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.566562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.554138    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.670595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.670528    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.670953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.670926    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.671255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.671234    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.671465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.671430    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.671698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.671676    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.671906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.671875    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.672031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.671896    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssmdc\" (UniqueName: \"kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-kube-api-access-ssmdc\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.672290 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.672254    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.672480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.672461    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.672718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.672698    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.673525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.672857    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.673525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.670986    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.673525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.673419    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.676380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.676354    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.711640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.696129    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.828570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.810812    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.926635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.914021    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssmdc\" (UniqueName: \"kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-kube-api-access-ssmdc\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.969254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.969146    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:09.990092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:09.971128    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:10.520349 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:08:10.520320    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f881b2_0e66_4706_b9d1_4b1842b5d33c.slice/crio-0e4ba75b2cf75a06ff33ff87721c555e155a7bea9a53aaba8f6ea8ff0b4b6f7a WatchSource:0}: Error finding container 0e4ba75b2cf75a06ff33ff87721c555e155a7bea9a53aaba8f6ea8ff0b4b6f7a: Status 404 returned error can't find the container with id 0e4ba75b2cf75a06ff33ff87721c555e155a7bea9a53aaba8f6ea8ff0b4b6f7a
Oct 30 14:08:10.523114 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:10.523088    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"]
Oct 30 14:08:10.523271 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:10.523237    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:08:10.523355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:10.523319    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:08:10.523416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:10.523365    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:08:11.177339 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:11.177303    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" event={"ID":"48f881b2-0e66-4706-b9d1-4b1842b5d33c","Type":"ContainerStarted","Data":"4407ac55a8048fb70d2ac3e32039152b6b76d39aa8edb1f00deede39942054a9"}
Oct 30 14:08:11.177505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:11.177361    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" event={"ID":"48f881b2-0e66-4706-b9d1-4b1842b5d33c","Type":"ContainerStarted","Data":"0e4ba75b2cf75a06ff33ff87721c555e155a7bea9a53aaba8f6ea8ff0b4b6f7a"}
Oct 30 14:08:11.251803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:11.251748    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" podStartSLOduration=2.251734504 podStartE2EDuration="2.251734504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:11.247698592 +0000 UTC m=+1156.279597537" watchObservedRunningTime="2026-10-30 14:08:11.251734504 +0000 UTC m=+1156.283633505"
Oct 30 14:08:11.366460 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:08:11.366420    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:08:11.972801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:11.972764    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:11.974248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:11.974219    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 14:08:11.974396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:11.974281    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" podUID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 14:08:12.975415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:12.975373    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 14:08:12.976185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:12.976135    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" podUID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 14:08:13.972786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:13.972751    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 14:08:13.972967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:13.972805    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" podUID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 14:08:14.972828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:14.972797    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 14:08:14.973255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:14.972849    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" podUID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 14:08:15.972336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:15.972299    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 14:08:15.972498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:15.972349    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" podUID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 14:08:16.972754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:16.972717    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 14:08:16.973180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:16.972781    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" podUID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 14:08:17.976510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:17.976481    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:17.976910 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:17.976775    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:17.977456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:17.977437    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:08:22.172125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:22.172090    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:08:23.545344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.545299    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"]
Oct 30 14:08:23.549522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.549502    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:23.577301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.568938    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"]
Oct 30 14:08:23.577829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.577803    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:08:23.578015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.577960    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:08:23.725341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.725309    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8521543e-z5kdg\" (UID: \"2a53caf1-226d-40b7-be45-31c6d59795a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:23.725507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.725354    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wf2d\" (UniqueName: \"kubernetes.io/projected/2a53caf1-226d-40b7-be45-31c6d59795a0-kube-api-access-4wf2d\") pod \"maas-api-revoke-sub-8521543e-z5kdg\" (UID: \"2a53caf1-226d-40b7-be45-31c6d59795a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:23.725507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.725487    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8521543e-z5kdg\" (UID: \"2a53caf1-226d-40b7-be45-31c6d59795a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:23.827117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.827088    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8521543e-z5kdg\" (UID: \"2a53caf1-226d-40b7-be45-31c6d59795a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:23.827263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.827130    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wf2d\" (UniqueName: \"kubernetes.io/projected/2a53caf1-226d-40b7-be45-31c6d59795a0-kube-api-access-4wf2d\") pod \"maas-api-revoke-sub-8521543e-z5kdg\" (UID: \"2a53caf1-226d-40b7-be45-31c6d59795a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:23.827263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.827180    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8521543e-z5kdg\" (UID: \"2a53caf1-226d-40b7-be45-31c6d59795a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:23.827743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.827718    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8521543e-z5kdg\" (UID: \"2a53caf1-226d-40b7-be45-31c6d59795a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:23.829627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.829608    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8521543e-z5kdg\" (UID: \"2a53caf1-226d-40b7-be45-31c6d59795a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:23.854094 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.854066    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wf2d\" (UniqueName: \"kubernetes.io/projected/2a53caf1-226d-40b7-be45-31c6d59795a0-kube-api-access-4wf2d\") pod \"maas-api-revoke-sub-8521543e-z5kdg\" (UID: \"2a53caf1-226d-40b7-be45-31c6d59795a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:23.863741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:23.863720    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:24.164115 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:08:24.164082    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a53caf1_226d_40b7_be45_31c6d59795a0.slice/crio-91e9539d5593f213b01e5be8647b9284252c036b64c3b6043751ff3d4e790aa8 WatchSource:0}: Error finding container 91e9539d5593f213b01e5be8647b9284252c036b64c3b6043751ff3d4e790aa8: Status 404 returned error can't find the container with id 91e9539d5593f213b01e5be8647b9284252c036b64c3b6043751ff3d4e790aa8
Oct 30 14:08:24.166267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.166241    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"]
Oct 30 14:08:24.298335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.298306    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg" event={"ID":"2a53caf1-226d-40b7-be45-31c6d59795a0","Type":"ContainerStarted","Data":"91e9539d5593f213b01e5be8647b9284252c036b64c3b6043751ff3d4e790aa8"}
Oct 30 14:08:24.542142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.542115    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"]
Oct 30 14:08:24.547146 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.547120    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.574378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.574349    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"]
Oct 30 14:08:24.635343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.635312    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72smx\" (UniqueName: \"kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-kube-api-access-72smx\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.635480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.635411    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.635480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.635446    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.635480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.635465    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb30f186-609d-4e05-9018-9ce4b63a133c-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.635640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.635619    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.635680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.635659    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.635756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.635743    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.635791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.635779    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.635827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.635800    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.737563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.736865    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb30f186-609d-4e05-9018-9ce4b63a133c-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.737563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.737041    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.737563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.737091    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.737563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.737148    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.737563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.737201    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.737563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.737261    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.737563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.737390    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72smx\" (UniqueName: \"kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-kube-api-access-72smx\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.737563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.737471    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.737563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.737514    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.740647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.739120    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.740647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.739360    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.740647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.740270    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb30f186-609d-4e05-9018-9ce4b63a133c-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.740913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.740650    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.740980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.740916    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.744058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.744029    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.747435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.747410    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.765442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.765410    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.765764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.765745    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72smx\" (UniqueName: \"kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-kube-api-access-72smx\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:24.864735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:24.864704    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:25.187854 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:08:25.187828    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb30f186_609d_4e05_9018_9ce4b63a133c.slice/crio-3f0d9591ee75d304709616c2ae0d96aa6c58b9c288ef8cf68f069bbba50d3342 WatchSource:0}: Error finding container 3f0d9591ee75d304709616c2ae0d96aa6c58b9c288ef8cf68f069bbba50d3342: Status 404 returned error can't find the container with id 3f0d9591ee75d304709616c2ae0d96aa6c58b9c288ef8cf68f069bbba50d3342
Oct 30 14:08:25.190353 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:25.190317    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:08:25.190471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:25.190398    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:08:25.190471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:25.190444    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:08:25.191267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:25.191239    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"]
Oct 30 14:08:25.308484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:25.308450    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" event={"ID":"cb30f186-609d-4e05-9018-9ce4b63a133c","Type":"ContainerStarted","Data":"b11fd375802da1ba34ab1ec0f63f91e44d145e2a6c0db8ec0409b6b122581a12"}
Oct 30 14:08:25.308685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:25.308513    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" event={"ID":"cb30f186-609d-4e05-9018-9ce4b63a133c","Type":"ContainerStarted","Data":"3f0d9591ee75d304709616c2ae0d96aa6c58b9c288ef8cf68f069bbba50d3342"}
Oct 30 14:08:25.310406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:25.310374    4530 generic.go:356] "Generic (PLEG): container finished" podID="2a53caf1-226d-40b7-be45-31c6d59795a0" containerID="db87d9600234a251759006bb730c1f9c116cf36bcca453849b4001c443851a68" exitCode=0
Oct 30 14:08:25.310597 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:25.310450    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg" event={"ID":"2a53caf1-226d-40b7-be45-31c6d59795a0","Type":"ContainerDied","Data":"db87d9600234a251759006bb730c1f9c116cf36bcca453849b4001c443851a68"}
Oct 30 14:08:25.362700 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:25.362608    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" podStartSLOduration=1.362594528 podStartE2EDuration="1.362594528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:25.358914915 +0000 UTC m=+1170.390813867" watchObservedRunningTime="2026-10-30 14:08:25.362594528 +0000 UTC m=+1170.394493467"
Oct 30 14:08:25.865667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:25.865634    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:26.502844 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.502805    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:26.559337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.559307    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a53caf1-226d-40b7-be45-31c6d59795a0-kube-api-access-4wf2d\" (UniqueName: \"kubernetes.io/projected/2a53caf1-226d-40b7-be45-31c6d59795a0-kube-api-access-4wf2d\") pod \"2a53caf1-226d-40b7-be45-31c6d59795a0\" (UID: \"2a53caf1-226d-40b7-be45-31c6d59795a0\") "
Oct 30 14:08:26.559496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.559356    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-xks-ca\") pod \"2a53caf1-226d-40b7-be45-31c6d59795a0\" (UID: \"2a53caf1-226d-40b7-be45-31c6d59795a0\") "
Oct 30 14:08:26.559496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.559440    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-openshift-ca\") pod \"2a53caf1-226d-40b7-be45-31c6d59795a0\" (UID: \"2a53caf1-226d-40b7-be45-31c6d59795a0\") "
Oct 30 14:08:26.559841 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.559808    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-openshift-ca" pod "2a53caf1-226d-40b7-be45-31c6d59795a0" (UID: "2a53caf1-226d-40b7-be45-31c6d59795a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:26.560254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.560231    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:08:26.561820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.561789    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a53caf1-226d-40b7-be45-31c6d59795a0-kube-api-access-4wf2d" pod "2a53caf1-226d-40b7-be45-31c6d59795a0" (UID: "2a53caf1-226d-40b7-be45-31c6d59795a0"). InnerVolumeSpecName "kube-api-access-4wf2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:26.562209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.562186    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-xks-ca" pod "2a53caf1-226d-40b7-be45-31c6d59795a0" (UID: "2a53caf1-226d-40b7-be45-31c6d59795a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:08:26.674276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.671017    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wf2d\" (UniqueName: \"kubernetes.io/projected/2a53caf1-226d-40b7-be45-31c6d59795a0-kube-api-access-4wf2d\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:08:26.674276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.671054    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a53caf1-226d-40b7-be45-31c6d59795a0-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:08:26.750743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.750709    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"]
Oct 30 14:08:26.751808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.751785    4530 cpu_manager.go:414] "containerMap: removing container" podUID="2a53caf1-226d-40b7-be45-31c6d59795a0" containerName="revoke-keys"
Oct 30 14:08:26.751983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.751969    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a53caf1-226d-40b7-be45-31c6d59795a0" containerName="revoke-keys"
Oct 30 14:08:26.752177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.752168    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a53caf1-226d-40b7-be45-31c6d59795a0" containerName="revoke-keys"
Oct 30 14:08:26.756107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.756079    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:26.757961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.757931    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"]
Oct 30 14:08:26.773039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.772250    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjp8\" (UniqueName: \"kubernetes.io/projected/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-kube-api-access-sfjp8\") pod \"maas-api-revoke-sub-d08a9267-9btxg\" (UID: \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:26.773039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.772320    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d08a9267-9btxg\" (UID: \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:26.773039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.772471    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d08a9267-9btxg\" (UID: \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:26.866347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.866275    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 14:08:26.866347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.866321    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" podUID="cb30f186-609d-4e05-9018-9ce4b63a133c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 14:08:26.873963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.873928    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjp8\" (UniqueName: \"kubernetes.io/projected/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-kube-api-access-sfjp8\") pod \"maas-api-revoke-sub-d08a9267-9btxg\" (UID: \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:26.874068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.873990    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d08a9267-9btxg\" (UID: \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:26.874068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.874047    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d08a9267-9btxg\" (UID: \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:26.874630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.874611    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d08a9267-9btxg\" (UID: \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:26.876257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.876222    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d08a9267-9btxg\" (UID: \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:26.907972 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:26.907942    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjp8\" (UniqueName: \"kubernetes.io/projected/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-kube-api-access-sfjp8\") pod \"maas-api-revoke-sub-d08a9267-9btxg\" (UID: \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:27.076052 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:27.076019    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:27.326463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:27.326434    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"
Oct 30 14:08:27.326659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:27.326475    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg" event={"ID":"2a53caf1-226d-40b7-be45-31c6d59795a0","Type":"ContainerDied","Data":"91e9539d5593f213b01e5be8647b9284252c036b64c3b6043751ff3d4e790aa8"}
Oct 30 14:08:27.326659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:27.326526    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91e9539d5593f213b01e5be8647b9284252c036b64c3b6043751ff3d4e790aa8"
Oct 30 14:08:27.409524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:27.409442    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"]
Oct 30 14:08:27.410038 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:08:27.410007    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd8ceea_0d38_4570_bab7_0cdd57f8723a.slice/crio-165db58b9d8b8b2cc7234cc49eda44ed81baa9c8b1fe386315e2394c0ab324af WatchSource:0}: Error finding container 165db58b9d8b8b2cc7234cc49eda44ed81baa9c8b1fe386315e2394c0ab324af: Status 404 returned error can't find the container with id 165db58b9d8b8b2cc7234cc49eda44ed81baa9c8b1fe386315e2394c0ab324af
Oct 30 14:08:27.865999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:27.865968    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 14:08:27.866132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:27.866012    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" podUID="cb30f186-609d-4e05-9018-9ce4b63a133c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 14:08:28.334805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:28.334758    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg" event={"ID":"fcd8ceea-0d38-4570-bab7-0cdd57f8723a","Type":"ContainerStarted","Data":"ee6b3bc0d65bb574d390d06a34cc2f284e0a80600c37df6168cb714d9f96be0b"}
Oct 30 14:08:28.335164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:28.334823    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg" event={"ID":"fcd8ceea-0d38-4570-bab7-0cdd57f8723a","Type":"ContainerStarted","Data":"165db58b9d8b8b2cc7234cc49eda44ed81baa9c8b1fe386315e2394c0ab324af"}
Oct 30 14:08:28.865981 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:28.865941    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 14:08:28.866154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:28.865993    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" podUID="cb30f186-609d-4e05-9018-9ce4b63a133c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 14:08:29.344142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:29.344105    4530 generic.go:356] "Generic (PLEG): container finished" podID="fcd8ceea-0d38-4570-bab7-0cdd57f8723a" containerID="ee6b3bc0d65bb574d390d06a34cc2f284e0a80600c37df6168cb714d9f96be0b" exitCode=0
Oct 30 14:08:29.344512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:29.344184    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg" event={"ID":"fcd8ceea-0d38-4570-bab7-0cdd57f8723a","Type":"ContainerDied","Data":"ee6b3bc0d65bb574d390d06a34cc2f284e0a80600c37df6168cb714d9f96be0b"}
Oct 30 14:08:29.865445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:29.865412    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 14:08:29.865622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:29.865465    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" podUID="cb30f186-609d-4e05-9018-9ce4b63a133c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 14:08:30.546444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:30.546418    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:30.650302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:30.650226    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-openshift-ca\") pod \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\" (UID: \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\") "
Oct 30 14:08:30.650443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:30.650310    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-xks-ca\") pod \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\" (UID: \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\") "
Oct 30 14:08:30.650443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:30.650420    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-kube-api-access-sfjp8\" (UniqueName: \"kubernetes.io/projected/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-kube-api-access-sfjp8\") pod \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\" (UID: \"fcd8ceea-0d38-4570-bab7-0cdd57f8723a\") "
Oct 30 14:08:30.650677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:30.650636    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-openshift-ca" pod "fcd8ceea-0d38-4570-bab7-0cdd57f8723a" (UID: "fcd8ceea-0d38-4570-bab7-0cdd57f8723a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:30.650942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:30.650916    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:08:30.652627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:30.652601    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-kube-api-access-sfjp8" pod "fcd8ceea-0d38-4570-bab7-0cdd57f8723a" (UID: "fcd8ceea-0d38-4570-bab7-0cdd57f8723a"). InnerVolumeSpecName "kube-api-access-sfjp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:30.652709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:30.652663    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-xks-ca" pod "fcd8ceea-0d38-4570-bab7-0cdd57f8723a" (UID: "fcd8ceea-0d38-4570-bab7-0cdd57f8723a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:08:30.752744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:30.752712    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:08:30.752901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:30.752740    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfjp8\" (UniqueName: \"kubernetes.io/projected/fcd8ceea-0d38-4570-bab7-0cdd57f8723a-kube-api-access-sfjp8\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:08:30.865441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:30.865199    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 14:08:30.865908 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:30.865860    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" podUID="cb30f186-609d-4e05-9018-9ce4b63a133c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 14:08:31.360173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:31.360146    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"
Oct 30 14:08:31.360316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:31.360276    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg" event={"ID":"fcd8ceea-0d38-4570-bab7-0cdd57f8723a","Type":"ContainerDied","Data":"165db58b9d8b8b2cc7234cc49eda44ed81baa9c8b1fe386315e2394c0ab324af"}
Oct 30 14:08:31.360377 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:31.360316    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="165db58b9d8b8b2cc7234cc49eda44ed81baa9c8b1fe386315e2394c0ab324af"
Oct 30 14:08:32.178651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:32.178559    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:32.179165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:32.179131    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:32.183250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:32.183225    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:08:39.581820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.581786    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"]
Oct 30 14:08:39.582361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.582348    4530 cpu_manager.go:414] "containerMap: removing container" podUID="fcd8ceea-0d38-4570-bab7-0cdd57f8723a" containerName="revoke-keys"
Oct 30 14:08:39.582361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.582361    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd8ceea-0d38-4570-bab7-0cdd57f8723a" containerName="revoke-keys"
Oct 30 14:08:39.582482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.582473    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd8ceea-0d38-4570-bab7-0cdd57f8723a" containerName="revoke-keys"
Oct 30 14:08:39.586267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.586249    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.613402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.613373    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"]
Oct 30 14:08:39.760994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.760963    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.760994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.761004    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.761209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.761027    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.761209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.761191    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729aa207-6f8b-4283-8411-5bbd2eb20463-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.761290 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.761235    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.761327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.761291    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t47k6\" (UniqueName: \"kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-kube-api-access-t47k6\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.761382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.761358    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.761462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.761438    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.761521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.761489    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.862951    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.862997    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.863026    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.863103    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729aa207-6f8b-4283-8411-5bbd2eb20463-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.863153    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.863284    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t47k6\" (UniqueName: \"kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-kube-api-access-t47k6\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.863334    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.863414    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.863441    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.863472    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.863772    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.863804    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.863868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.863851    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.864301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.864272    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729aa207-6f8b-4283-8411-5bbd2eb20463-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.865392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.865374    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.865829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.865813    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.889435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.889411    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.889788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.889771    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t47k6\" (UniqueName: \"kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-kube-api-access-t47k6\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:39.899599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:39.899580    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:40.221533 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:08:40.221504    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729aa207_6f8b_4283_8411_5bbd2eb20463.slice/crio-41933d5f202d16daee46b35564e77d998e69b267f87c9b6495c586533e3c5268 WatchSource:0}: Error finding container 41933d5f202d16daee46b35564e77d998e69b267f87c9b6495c586533e3c5268: Status 404 returned error can't find the container with id 41933d5f202d16daee46b35564e77d998e69b267f87c9b6495c586533e3c5268
Oct 30 14:08:40.222334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.222310    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"]
Oct 30 14:08:40.224031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.223999    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:08:40.224114 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.224061    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:08:40.224114 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.224089    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:08:40.502573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.502446    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" event={"ID":"729aa207-6f8b-4283-8411-5bbd2eb20463","Type":"ContainerStarted","Data":"a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3"}
Oct 30 14:08:40.502573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.502503    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" event={"ID":"729aa207-6f8b-4283-8411-5bbd2eb20463","Type":"ContainerStarted","Data":"41933d5f202d16daee46b35564e77d998e69b267f87c9b6495c586533e3c5268"}
Oct 30 14:08:40.562103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.562051    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" podStartSLOduration=1.562034667 podStartE2EDuration="1.562034667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:40.560136474 +0000 UTC m=+1185.592035439" watchObservedRunningTime="2026-10-30 14:08:40.562034667 +0000 UTC m=+1185.593933606"
Oct 30 14:08:40.703361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.703328    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"]
Oct 30 14:08:40.708882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.708854    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.723831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.723794    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"]
Oct 30 14:08:40.732303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.732258    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"kube-root-ca.crt\""
Oct 30 14:08:40.732303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.732269    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"openshift-service-ca.crt\""
Oct 30 14:08:40.732717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.732700    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"default-dockercfg-hskdp\""
Oct 30 14:08:40.732796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.732754    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"distinct-w2-da78c4-kserve-self-signed-certs\""
Oct 30 14:08:40.792888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.783293    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hsn2\" (UniqueName: \"kubernetes.io/projected/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-kube-api-access-8hsn2\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.792888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.783360    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tls-certs\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.792888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.783408    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-home\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.792888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.783429    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-dshm\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.792888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.783507    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tmp-dir\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.792888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.783534    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-model-cache\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.908585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.900641    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:40.919570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.911484    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hsn2\" (UniqueName: \"kubernetes.io/projected/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-kube-api-access-8hsn2\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.919570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.911579    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tls-certs\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.919570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.911640    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-home\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.919570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.911662    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-dshm\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.919570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.911771    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tmp-dir\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.919570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.911798    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-model-cache\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.919570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.912445    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-model-cache\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.919570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.913007    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-home\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.919570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.913701    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tmp-dir\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.960566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.960079    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tls-certs\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.960566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.960335    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-dshm\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:40.981570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:40.974501    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hsn2\" (UniqueName: \"kubernetes.io/projected/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-kube-api-access-8hsn2\") pod \"distinct-w2-da78c4-kserve-5bf54cc768-v7pk8\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:41.030266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:41.028836    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:41.412582 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:08:41.412253    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:08:41.609321 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:41.609289    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"]
Oct 30 14:08:41.609861 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:08:41.609836    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod276cdf23_88b0_4fc2_a20c_c5c4c27db30f.slice/crio-1f3dbf0471a79e28717f1354174224a6ba19ae5d826d0dde0628b80c82802ceb WatchSource:0}: Error finding container 1f3dbf0471a79e28717f1354174224a6ba19ae5d826d0dde0628b80c82802ceb: Status 404 returned error can't find the container with id 1f3dbf0471a79e28717f1354174224a6ba19ae5d826d0dde0628b80c82802ceb
Oct 30 14:08:41.906285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:41.906205    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 14:08:41.906285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:41.906254    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 14:08:42.520103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:42.520064    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" event={"ID":"276cdf23-88b0-4fc2-a20c-c5c4c27db30f","Type":"ContainerStarted","Data":"fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4"}
Oct 30 14:08:42.520252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:42.520111    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" event={"ID":"276cdf23-88b0-4fc2-a20c-c5c4c27db30f","Type":"ContainerStarted","Data":"1f3dbf0471a79e28717f1354174224a6ba19ae5d826d0dde0628b80c82802ceb"}
Oct 30 14:08:42.569478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:42.569432    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podStartSLOduration=2.569417994 podStartE2EDuration="2.569417994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:42.566690083 +0000 UTC m=+1187.598589022" watchObservedRunningTime="2026-10-30 14:08:42.569417994 +0000 UTC m=+1187.601316932"
Oct 30 14:08:42.904934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:42.904889    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 14:08:42.905106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:42.904963    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 14:08:43.036578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:43.030785    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:43.723970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:43.723935    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:08:43.904884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:43.904850    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 14:08:43.905092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:43.904912    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 14:08:44.901130 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:44.901097    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 14:08:44.901794 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:44.901744    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 14:08:45.904954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:45.904912    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 14:08:45.905373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:45.904975    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 14:08:46.900373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:46.900339    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 14:08:46.900570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:46.900391    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 14:08:47.904437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:47.904407    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:47.904887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:47.904869    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:47.905458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:47.905441    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:08:51.029603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:51.029566    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:51.041913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:51.041887    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:51.693803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:51.693772    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:08:54.951644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:54.951608    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"]
Oct 30 14:08:55.031976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.031941    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.056230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.056205    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"]
Oct 30 14:08:55.248462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.248365    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.249800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.249771    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.250206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.250185    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmmbb\" (UniqueName: \"kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-kube-api-access-xmmbb\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.250428 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.250409    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.262949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.262915    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.263068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.262988    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.263068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.263042    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.263202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.263073    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.263202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.263100    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.364626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.364588    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.364777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.364637    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.364777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.364683    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.364777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.364744    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.364777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.364770    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.364918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.364811    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.364918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.364903    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.365086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.365011    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.365163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.365143    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmmbb\" (UniqueName: \"kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-kube-api-access-xmmbb\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.365500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.365480    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.365858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.365837    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.366354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.366337    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.366777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.366760    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.367282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.367265    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.368957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.368932    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.369421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.369393    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.380836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.380814    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:08:55.381577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.381558    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:08:55.393581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.393179    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.396469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.396142    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmmbb\" (UniqueName: \"kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-kube-api-access-xmmbb\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:55.397293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.397110    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:08:55.398067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.397996    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:08:55.399625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.399241    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:08:55.401195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.401110    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:08:55.663241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:55.663207    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:56.147203 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:08:56.147167    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd5bb1a_fceb_4ca4_997a_e197ea4277c1.slice/crio-31af9344ccb8a00bc5076d31e9253055790e89520ca101c48d1dc1406904739c WatchSource:0}: Error finding container 31af9344ccb8a00bc5076d31e9253055790e89520ca101c48d1dc1406904739c: Status 404 returned error can't find the container with id 31af9344ccb8a00bc5076d31e9253055790e89520ca101c48d1dc1406904739c
Oct 30 14:08:56.149616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:56.149593    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"]
Oct 30 14:08:56.150329 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:56.150295    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:08:56.150399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:56.150364    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:08:56.150442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:56.150409    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:08:56.790565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:56.779516    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" event={"ID":"acd5bb1a-fceb-4ca4-997a-e197ea4277c1","Type":"ContainerStarted","Data":"3ec2a4ca4db28e38a56b9d336a3805c55c343615128f9eac27b87ded2d7ca17a"}
Oct 30 14:08:56.790565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:56.779621    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" event={"ID":"acd5bb1a-fceb-4ca4-997a-e197ea4277c1","Type":"ContainerStarted","Data":"31af9344ccb8a00bc5076d31e9253055790e89520ca101c48d1dc1406904739c"}
Oct 30 14:08:56.869714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:56.869663    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" podStartSLOduration=2.869649244 podStartE2EDuration="2.869649244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:56.851253883 +0000 UTC m=+1201.883152861" watchObservedRunningTime="2026-10-30 14:08:56.869649244 +0000 UTC m=+1201.901548182"
Oct 30 14:08:57.647773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.647738    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"]
Oct 30 14:08:57.652023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.652002    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:08:57.664380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.664330    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:08:57.668644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.668619    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 14:08:57.668760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.668666    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 14:08:57.676345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.676317    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"]
Oct 30 14:08:57.698564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.695880    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:08:57.698564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.696004    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:08:57.718516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.718486    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad9f7071-fn5z9\" (UID: \"e085efcd-2a7e-4798-adf0-fdb57b468308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:08:57.719007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.718980    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad9f7071-fn5z9\" (UID: \"e085efcd-2a7e-4798-adf0-fdb57b468308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:08:57.719213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.719195    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4sr\" (UniqueName: \"kubernetes.io/projected/e085efcd-2a7e-4798-adf0-fdb57b468308-kube-api-access-mk4sr\") pod \"maas-api-revoke-sub-ad9f7071-fn5z9\" (UID: \"e085efcd-2a7e-4798-adf0-fdb57b468308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:08:57.820779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.820742    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad9f7071-fn5z9\" (UID: \"e085efcd-2a7e-4798-adf0-fdb57b468308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:08:57.820967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.820837    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4sr\" (UniqueName: \"kubernetes.io/projected/e085efcd-2a7e-4798-adf0-fdb57b468308-kube-api-access-mk4sr\") pod \"maas-api-revoke-sub-ad9f7071-fn5z9\" (UID: \"e085efcd-2a7e-4798-adf0-fdb57b468308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:08:57.820967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.820908    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad9f7071-fn5z9\" (UID: \"e085efcd-2a7e-4798-adf0-fdb57b468308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:08:57.821646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.821623    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad9f7071-fn5z9\" (UID: \"e085efcd-2a7e-4798-adf0-fdb57b468308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:08:57.823112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.823092    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad9f7071-fn5z9\" (UID: \"e085efcd-2a7e-4798-adf0-fdb57b468308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:08:57.847919 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.847895    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4sr\" (UniqueName: \"kubernetes.io/projected/e085efcd-2a7e-4798-adf0-fdb57b468308-kube-api-access-mk4sr\") pod \"maas-api-revoke-sub-ad9f7071-fn5z9\" (UID: \"e085efcd-2a7e-4798-adf0-fdb57b468308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:08:57.967239 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:57.967157    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:08:58.307637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:58.307589    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"]
Oct 30 14:08:58.314115 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:08:58.314081    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode085efcd_2a7e_4798_adf0_fdb57b468308.slice/crio-4a730179750d371c282c25ebbb34f216e62f0e579d4267729644b64664a33481 WatchSource:0}: Error finding container 4a730179750d371c282c25ebbb34f216e62f0e579d4267729644b64664a33481: Status 404 returned error can't find the container with id 4a730179750d371c282c25ebbb34f216e62f0e579d4267729644b64664a33481
Oct 30 14:08:58.664265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:58.664190    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 14:08:58.664265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:58.664245    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 14:08:58.824485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:58.824453    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9" event={"ID":"e085efcd-2a7e-4798-adf0-fdb57b468308","Type":"ContainerStarted","Data":"7ffb8e96fd80b82039b9a0d01e0cf9a31530dd38e2897e8dd55e4d8ff346f548"}
Oct 30 14:08:58.824668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:58.824505    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9" event={"ID":"e085efcd-2a7e-4798-adf0-fdb57b468308","Type":"ContainerStarted","Data":"4a730179750d371c282c25ebbb34f216e62f0e579d4267729644b64664a33481"}
Oct 30 14:08:58.872419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:58.872376    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9" podStartSLOduration=1.872364389 podStartE2EDuration="1.872364389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:58.869220893 +0000 UTC m=+1203.901119831" watchObservedRunningTime="2026-10-30 14:08:58.872364389 +0000 UTC m=+1203.904263327"
Oct 30 14:08:59.664333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:59.664252    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 14:08:59.664333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:59.664302    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 14:08:59.833654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:59.833622    4530 generic.go:356] "Generic (PLEG): container finished" podID="e085efcd-2a7e-4798-adf0-fdb57b468308" containerID="7ffb8e96fd80b82039b9a0d01e0cf9a31530dd38e2897e8dd55e4d8ff346f548" exitCode=0
Oct 30 14:08:59.833831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:08:59.833709    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9" event={"ID":"e085efcd-2a7e-4798-adf0-fdb57b468308","Type":"ContainerDied","Data":"7ffb8e96fd80b82039b9a0d01e0cf9a31530dd38e2897e8dd55e4d8ff346f548"}
Oct 30 14:09:00.663882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:00.663846    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 14:09:00.664068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:00.663898    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 14:09:01.049922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.049894    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:09:01.111252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.111220    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e085efcd-2a7e-4798-adf0-fdb57b468308-kube-api-access-mk4sr\" (UniqueName: \"kubernetes.io/projected/e085efcd-2a7e-4798-adf0-fdb57b468308-kube-api-access-mk4sr\") pod \"e085efcd-2a7e-4798-adf0-fdb57b468308\" (UID: \"e085efcd-2a7e-4798-adf0-fdb57b468308\") "
Oct 30 14:09:01.111442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.111309    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-xks-ca\") pod \"e085efcd-2a7e-4798-adf0-fdb57b468308\" (UID: \"e085efcd-2a7e-4798-adf0-fdb57b468308\") "
Oct 30 14:09:01.111442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.111372    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-openshift-ca\") pod \"e085efcd-2a7e-4798-adf0-fdb57b468308\" (UID: \"e085efcd-2a7e-4798-adf0-fdb57b468308\") "
Oct 30 14:09:01.112043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.112015    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-openshift-ca" pod "e085efcd-2a7e-4798-adf0-fdb57b468308" (UID: "e085efcd-2a7e-4798-adf0-fdb57b468308"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:09:01.116205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.113805    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e085efcd-2a7e-4798-adf0-fdb57b468308-kube-api-access-mk4sr" pod "e085efcd-2a7e-4798-adf0-fdb57b468308" (UID: "e085efcd-2a7e-4798-adf0-fdb57b468308"). InnerVolumeSpecName "kube-api-access-mk4sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:01.116205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.115780    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-xks-ca" pod "e085efcd-2a7e-4798-adf0-fdb57b468308" (UID: "e085efcd-2a7e-4798-adf0-fdb57b468308"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:01.213669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.213587    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:01.213669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.213616    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk4sr\" (UniqueName: \"kubernetes.io/projected/e085efcd-2a7e-4798-adf0-fdb57b468308-kube-api-access-mk4sr\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:01.213669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.213626    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e085efcd-2a7e-4798-adf0-fdb57b468308-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:01.663737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.663701    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 14:09:01.663906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.663766    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 14:09:01.851843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.851809    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9" event={"ID":"e085efcd-2a7e-4798-adf0-fdb57b468308","Type":"ContainerDied","Data":"4a730179750d371c282c25ebbb34f216e62f0e579d4267729644b64664a33481"}
Oct 30 14:09:01.851987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.851855    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a730179750d371c282c25ebbb34f216e62f0e579d4267729644b64664a33481"
Oct 30 14:09:01.851987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:01.851818    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"
Oct 30 14:09:02.664474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:02.664442    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 14:09:02.664987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:02.664492    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 14:09:03.668050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:03.668026    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:09:03.668469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:03.668404    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:09:03.669009 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:03.668990    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:09:04.779864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:04.779829    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:09:08.473989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.473891    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"]
Oct 30 14:09:08.474524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.474508    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e085efcd-2a7e-4798-adf0-fdb57b468308" containerName="revoke-keys"
Oct 30 14:09:08.474524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.474522    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e085efcd-2a7e-4798-adf0-fdb57b468308" containerName="revoke-keys"
Oct 30 14:09:08.474694 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.474648    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e085efcd-2a7e-4798-adf0-fdb57b468308" containerName="revoke-keys"
Oct 30 14:09:08.478622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.478597    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:08.502838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.502795    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"]
Oct 30 14:09:08.528979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.528779    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:09:08.528979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.528921    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:09:08.610350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.610316    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a1d377d-jgv8p\" (UID: \"64431e73-3911-42d5-b91e-89930c21b8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:08.610798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.610771    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94nqc\" (UniqueName: \"kubernetes.io/projected/64431e73-3911-42d5-b91e-89930c21b8aa-kube-api-access-94nqc\") pod \"maas-api-revoke-sub-2a1d377d-jgv8p\" (UID: \"64431e73-3911-42d5-b91e-89930c21b8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:08.611126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.611107    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a1d377d-jgv8p\" (UID: \"64431e73-3911-42d5-b91e-89930c21b8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:08.712465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.712422    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94nqc\" (UniqueName: \"kubernetes.io/projected/64431e73-3911-42d5-b91e-89930c21b8aa-kube-api-access-94nqc\") pod \"maas-api-revoke-sub-2a1d377d-jgv8p\" (UID: \"64431e73-3911-42d5-b91e-89930c21b8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:08.712678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.712604    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a1d377d-jgv8p\" (UID: \"64431e73-3911-42d5-b91e-89930c21b8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:08.712749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.712697    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a1d377d-jgv8p\" (UID: \"64431e73-3911-42d5-b91e-89930c21b8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:08.713362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.713335    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a1d377d-jgv8p\" (UID: \"64431e73-3911-42d5-b91e-89930c21b8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:08.714954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.714932    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a1d377d-jgv8p\" (UID: \"64431e73-3911-42d5-b91e-89930c21b8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:08.739819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.739766    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94nqc\" (UniqueName: \"kubernetes.io/projected/64431e73-3911-42d5-b91e-89930c21b8aa-kube-api-access-94nqc\") pod \"maas-api-revoke-sub-2a1d377d-jgv8p\" (UID: \"64431e73-3911-42d5-b91e-89930c21b8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:08.859984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:08.859956    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:09.166614 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:09:09.166587    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64431e73_3911_42d5_b91e_89930c21b8aa.slice/crio-d471e250bd9d04f7112d8ad7a487239a4ca9e379967cb1895b2d0df962f3f333 WatchSource:0}: Error finding container d471e250bd9d04f7112d8ad7a487239a4ca9e379967cb1895b2d0df962f3f333: Status 404 returned error can't find the container with id d471e250bd9d04f7112d8ad7a487239a4ca9e379967cb1895b2d0df962f3f333
Oct 30 14:09:09.169335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:09.169311    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"]
Oct 30 14:09:10.002024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:10.001985    4530 generic.go:356] "Generic (PLEG): container finished" podID="64431e73-3911-42d5-b91e-89930c21b8aa" containerID="9015b8779589504dff52b7d6dcbaee0418b98e1f9e859adc69ba3adaf93fb11e" exitCode=0
Oct 30 14:09:10.002460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:10.002180    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p" event={"ID":"64431e73-3911-42d5-b91e-89930c21b8aa","Type":"ContainerDied","Data":"9015b8779589504dff52b7d6dcbaee0418b98e1f9e859adc69ba3adaf93fb11e"}
Oct 30 14:09:10.002460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:10.002297    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p" event={"ID":"64431e73-3911-42d5-b91e-89930c21b8aa","Type":"ContainerStarted","Data":"d471e250bd9d04f7112d8ad7a487239a4ca9e379967cb1895b2d0df962f3f333"}
Oct 30 14:09:11.180222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:11.180197    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:11.250790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:11.250763    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-openshift-ca\") pod \"64431e73-3911-42d5-b91e-89930c21b8aa\" (UID: \"64431e73-3911-42d5-b91e-89930c21b8aa\") "
Oct 30 14:09:11.250958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:11.250945    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-xks-ca\") pod \"64431e73-3911-42d5-b91e-89930c21b8aa\" (UID: \"64431e73-3911-42d5-b91e-89930c21b8aa\") "
Oct 30 14:09:11.251003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:11.250982    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64431e73-3911-42d5-b91e-89930c21b8aa-kube-api-access-94nqc\" (UniqueName: \"kubernetes.io/projected/64431e73-3911-42d5-b91e-89930c21b8aa-kube-api-access-94nqc\") pod \"64431e73-3911-42d5-b91e-89930c21b8aa\" (UID: \"64431e73-3911-42d5-b91e-89930c21b8aa\") "
Oct 30 14:09:11.251096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:11.251074    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-openshift-ca" pod "64431e73-3911-42d5-b91e-89930c21b8aa" (UID: "64431e73-3911-42d5-b91e-89930c21b8aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:09:11.252097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:11.252069    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:11.253555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:11.253523    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-xks-ca" pod "64431e73-3911-42d5-b91e-89930c21b8aa" (UID: "64431e73-3911-42d5-b91e-89930c21b8aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:11.253654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:11.253581    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64431e73-3911-42d5-b91e-89930c21b8aa-kube-api-access-94nqc" pod "64431e73-3911-42d5-b91e-89930c21b8aa" (UID: "64431e73-3911-42d5-b91e-89930c21b8aa"). InnerVolumeSpecName "kube-api-access-94nqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:11.352808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:11.352784    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64431e73-3911-42d5-b91e-89930c21b8aa-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:11.352808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:11.352808    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94nqc\" (UniqueName: \"kubernetes.io/projected/64431e73-3911-42d5-b91e-89930c21b8aa-kube-api-access-94nqc\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:11.366277 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:09:11.366245    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:09:12.029560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:12.027748    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"
Oct 30 14:09:12.029560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:12.027772    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p" event={"ID":"64431e73-3911-42d5-b91e-89930c21b8aa","Type":"ContainerDied","Data":"d471e250bd9d04f7112d8ad7a487239a4ca9e379967cb1895b2d0df962f3f333"}
Oct 30 14:09:12.029560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:12.027819    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d471e250bd9d04f7112d8ad7a487239a4ca9e379967cb1895b2d0df962f3f333"
Oct 30 14:09:18.229442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.229402    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"]
Oct 30 14:09:18.230246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.230222    4530 cpu_manager.go:414] "containerMap: removing container" podUID="64431e73-3911-42d5-b91e-89930c21b8aa" containerName="revoke-keys"
Oct 30 14:09:18.230358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.230247    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64431e73-3911-42d5-b91e-89930c21b8aa" containerName="revoke-keys"
Oct 30 14:09:18.230453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.230437    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="64431e73-3911-42d5-b91e-89930c21b8aa" containerName="revoke-keys"
Oct 30 14:09:18.234040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.234016    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:18.257225 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.257200    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"]
Oct 30 14:09:18.263344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.263317    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:09:18.263480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.263388    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:09:18.322299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.322265    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ff8aa5-4xwww\" (UID: \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:18.322420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.322309    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcjp4\" (UniqueName: \"kubernetes.io/projected/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-kube-api-access-kcjp4\") pod \"maas-api-revoke-sub-62ff8aa5-4xwww\" (UID: \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:18.322497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.322477    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ff8aa5-4xwww\" (UID: \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:18.423443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.423414    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ff8aa5-4xwww\" (UID: \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:18.423662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.423643    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ff8aa5-4xwww\" (UID: \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:18.423709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.423689    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcjp4\" (UniqueName: \"kubernetes.io/projected/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-kube-api-access-kcjp4\") pod \"maas-api-revoke-sub-62ff8aa5-4xwww\" (UID: \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:18.424284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.424242    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ff8aa5-4xwww\" (UID: \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:18.426165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.426143    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ff8aa5-4xwww\" (UID: \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:18.450385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.450359    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcjp4\" (UniqueName: \"kubernetes.io/projected/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-kube-api-access-kcjp4\") pod \"maas-api-revoke-sub-62ff8aa5-4xwww\" (UID: \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:18.547309 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.547289    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:18.855481 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:09:18.855447    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d26f449_ff5d_4ba6_8269_4e16d2f2d41a.slice/crio-57e577ca39077f6493760469124926f83420496e251949074dd0789b85c7278d WatchSource:0}: Error finding container 57e577ca39077f6493760469124926f83420496e251949074dd0789b85c7278d: Status 404 returned error can't find the container with id 57e577ca39077f6493760469124926f83420496e251949074dd0789b85c7278d
Oct 30 14:09:18.857671 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:18.857645    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"]
Oct 30 14:09:19.102657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:19.102587    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww" event={"ID":"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a","Type":"ContainerStarted","Data":"57e577ca39077f6493760469124926f83420496e251949074dd0789b85c7278d"}
Oct 30 14:09:20.110814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:20.110784    4530 generic.go:356] "Generic (PLEG): container finished" podID="1d26f449-ff5d-4ba6-8269-4e16d2f2d41a" containerID="03d41d11e91f1097a8143120eb5b91e28323511ae8d66d445f12ef24678f43bb" exitCode=0
Oct 30 14:09:20.111210 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:20.110914    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww" event={"ID":"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a","Type":"ContainerDied","Data":"03d41d11e91f1097a8143120eb5b91e28323511ae8d66d445f12ef24678f43bb"}
Oct 30 14:09:21.269768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:21.269743    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:21.364417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:21.364385    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-kube-api-access-kcjp4\" (UniqueName: \"kubernetes.io/projected/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-kube-api-access-kcjp4\") pod \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\" (UID: \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\") "
Oct 30 14:09:21.364629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:21.364495    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-xks-ca\") pod \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\" (UID: \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\") "
Oct 30 14:09:21.364629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:21.364590    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-openshift-ca\") pod \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\" (UID: \"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a\") "
Oct 30 14:09:21.365185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:21.365152    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-openshift-ca" pod "1d26f449-ff5d-4ba6-8269-4e16d2f2d41a" (UID: "1d26f449-ff5d-4ba6-8269-4e16d2f2d41a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:09:21.366670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:21.366628    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-kube-api-access-kcjp4" pod "1d26f449-ff5d-4ba6-8269-4e16d2f2d41a" (UID: "1d26f449-ff5d-4ba6-8269-4e16d2f2d41a"). InnerVolumeSpecName "kube-api-access-kcjp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:21.366978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:21.366954    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-xks-ca" pod "1d26f449-ff5d-4ba6-8269-4e16d2f2d41a" (UID: "1d26f449-ff5d-4ba6-8269-4e16d2f2d41a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:21.466511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:21.466485    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcjp4\" (UniqueName: \"kubernetes.io/projected/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-kube-api-access-kcjp4\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:21.466511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:21.466508    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:21.466711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:21.466517    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:22.129027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:22.128996    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"
Oct 30 14:09:22.129228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:22.129028    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww" event={"ID":"1d26f449-ff5d-4ba6-8269-4e16d2f2d41a","Type":"ContainerDied","Data":"57e577ca39077f6493760469124926f83420496e251949074dd0789b85c7278d"}
Oct 30 14:09:22.129228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:22.129072    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57e577ca39077f6493760469124926f83420496e251949074dd0789b85c7278d"
Oct 30 14:09:25.194473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.194442    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"]
Oct 30 14:09:25.195070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.195052    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1d26f449-ff5d-4ba6-8269-4e16d2f2d41a" containerName="revoke-keys"
Oct 30 14:09:25.195070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.195069    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d26f449-ff5d-4ba6-8269-4e16d2f2d41a" containerName="revoke-keys"
Oct 30 14:09:25.195290 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.195276    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d26f449-ff5d-4ba6-8269-4e16d2f2d41a" containerName="revoke-keys"
Oct 30 14:09:25.225053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.225009    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.329452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.329310    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.329866 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.329838    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.330149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.330121    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.330613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.330591    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.330985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.330873    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.331249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.331139    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.331535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.331516    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7663d949-64c3-46ed-a16d-abf54a2b78ec-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.331919 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.331898    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjgt6\" (UniqueName: \"kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-kube-api-access-fjgt6\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.332592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.332568    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.363269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.363129    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"]
Oct 30 14:09:25.368171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.368143    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"]
Oct 30 14:09:25.373872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.373846    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.379785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.379757    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"]
Oct 30 14:09:25.434958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.434934    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.435067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.434981    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.435067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.435010    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.435067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.435027    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.435197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.435098    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.435197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.435126    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.435197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.435166    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.435375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.435346    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7663d949-64c3-46ed-a16d-abf54a2b78ec-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.435471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.435451    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgt6\" (UniqueName: \"kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-kube-api-access-fjgt6\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.435536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.435486    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.435536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.435509    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.435657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.435624    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.435657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.435628    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.436489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.436469    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7663d949-64c3-46ed-a16d-abf54a2b78ec-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.437470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.437447    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.437770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.437753    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.461688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.461630    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.462106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.462087    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjgt6\" (UniqueName: \"kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-kube-api-access-fjgt6\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.536608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.536585    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.536698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.536623    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.536762 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.536745    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.536796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.536789    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.536882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.536863    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.536976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.536963    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.537020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.537010    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.537059 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.537050    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qcxb\" (UniqueName: \"kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-kube-api-access-4qcxb\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.537092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.537075    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.609735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.609718    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:25.638436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.638412    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.638521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.638449    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.638521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.638510    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.638634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.638563    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.638634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.638618    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.638793 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.638775    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.638859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.638821    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.638917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.638860    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qcxb\" (UniqueName: \"kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-kube-api-access-4qcxb\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.638917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.638899    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.639043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.638989    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.639349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.639316    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.639432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.639395    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.639740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.639717    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.640199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.639335    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.641075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.641049    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.641447 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.641423    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.666170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.666142    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qcxb\" (UniqueName: \"kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-kube-api-access-4qcxb\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.666779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.666750    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.692836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.692812    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:25.933959 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:09:25.933931    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7663d949_64c3_46ed_a16d_abf54a2b78ec.slice/crio-22600474b58647dbd2ca1f0ca1fe8a123d75d30f7b345a5755cd1fc5d0172eb6 WatchSource:0}: Error finding container 22600474b58647dbd2ca1f0ca1fe8a123d75d30f7b345a5755cd1fc5d0172eb6: Status 404 returned error can't find the container with id 22600474b58647dbd2ca1f0ca1fe8a123d75d30f7b345a5755cd1fc5d0172eb6
Oct 30 14:09:25.936439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.936403    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:09:25.936578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.936458    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"]
Oct 30 14:09:25.936578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.936471    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:09:25.936578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:25.936501    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:09:26.012677 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:09:26.012648    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28a89c2_3e68_4cb3_a3ff_14a147b7e72b.slice/crio-1923d6d00444593dd9d11fb5c436540def6464a094470c1902471c823881f219 WatchSource:0}: Error finding container 1923d6d00444593dd9d11fb5c436540def6464a094470c1902471c823881f219: Status 404 returned error can't find the container with id 1923d6d00444593dd9d11fb5c436540def6464a094470c1902471c823881f219
Oct 30 14:09:26.015516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.015474    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:09:26.015655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.015576    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:09:26.015655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.015619    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:09:26.015744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.015703    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"]
Oct 30 14:09:26.143525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.143488    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:09:26.167081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.166984    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" event={"ID":"7663d949-64c3-46ed-a16d-abf54a2b78ec","Type":"ContainerStarted","Data":"861ae108b11109a183281056caf972fd1cfc63d394ec09e8156e3e63ba425344"}
Oct 30 14:09:26.167081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.167043    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" event={"ID":"7663d949-64c3-46ed-a16d-abf54a2b78ec","Type":"ContainerStarted","Data":"22600474b58647dbd2ca1f0ca1fe8a123d75d30f7b345a5755cd1fc5d0172eb6"}
Oct 30 14:09:26.170437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.170412    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" event={"ID":"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b","Type":"ContainerStarted","Data":"aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6"}
Oct 30 14:09:26.170618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.170459    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" event={"ID":"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b","Type":"ContainerStarted","Data":"1923d6d00444593dd9d11fb5c436540def6464a094470c1902471c823881f219"}
Oct 30 14:09:26.221854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.221798    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" podStartSLOduration=1.22178402 podStartE2EDuration="1.22178402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:26.219393845 +0000 UTC m=+1231.251292786" watchObservedRunningTime="2026-10-30 14:09:26.22178402 +0000 UTC m=+1231.253682959"
Oct 30 14:09:26.274799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.274694    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" podStartSLOduration=1.274674641 podStartE2EDuration="1.274674641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:26.269993902 +0000 UTC m=+1231.301892841" watchObservedRunningTime="2026-10-30 14:09:26.274674641 +0000 UTC m=+1231.306573573"
Oct 30 14:09:26.613564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.610657    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:26.695670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:26.695633    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:27.621562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:27.620745    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 14:09:27.621562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:27.620824    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 14:09:27.711578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:27.711464    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 14:09:27.711578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:27.711529    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 14:09:28.622021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:28.621985    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 14:09:28.622664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:28.622629    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 14:09:28.709418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:28.709379    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 14:09:28.709796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:28.709762    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 14:09:29.633627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:29.631645    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 14:09:29.633627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:29.631710    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 14:09:29.698152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:29.693326    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 14:09:29.698152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:29.693385    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 14:09:30.610566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:30.610287    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 14:09:30.610566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:30.610382    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 14:09:30.699914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:30.699873    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 14:09:30.700489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:30.700457    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 14:09:31.621895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:31.621857    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 14:09:31.622176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:31.622144    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 14:09:31.698572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:31.697890    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 14:09:31.698572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:31.697951    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 14:09:32.617015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:32.616976    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 14:09:32.617663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:32.617629    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 14:09:32.706565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:32.699107    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 14:09:32.706565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:32.699162    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 14:09:33.610719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:33.610685    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 14:09:33.610913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:33.610742    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 14:09:33.694168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:33.694128    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 14:09:33.694556 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:33.694200    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 14:09:35.668032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:35.667989    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:09:35.682720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:35.682660    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:09:35.756078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:35.756040    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:09:35.756395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:35.756358    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:09:36.119430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:36.119396    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:36.120240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:36.120215    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:36.131770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:36.131745    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:09:36.230335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:36.230307    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:36.230667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:36.230641    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:36.231641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:36.231619    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:09:39.809632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:39.809590    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"]
Oct 30 14:09:39.849535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:39.849503    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:39.912686 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:39.906264    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:09:39.912686 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:39.912632    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:09:39.990941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:39.990886    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f065f6cd-59qhv\" (UID: \"7a1df436-5c89-4c81-b77a-61580da3b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:39.991281 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:39.991256    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f065f6cd-59qhv\" (UID: \"7a1df436-5c89-4c81-b77a-61580da3b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:40.032643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:40.016890    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zpt\" (UniqueName: \"kubernetes.io/projected/7a1df436-5c89-4c81-b77a-61580da3b564-kube-api-access-m8zpt\") pod \"maas-api-revoke-sub-f065f6cd-59qhv\" (UID: \"7a1df436-5c89-4c81-b77a-61580da3b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:40.084846 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:40.084814    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"]
Oct 30 14:09:40.167515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:40.165785    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f065f6cd-59qhv\" (UID: \"7a1df436-5c89-4c81-b77a-61580da3b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:40.167515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:40.165879    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f065f6cd-59qhv\" (UID: \"7a1df436-5c89-4c81-b77a-61580da3b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:40.173562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:40.169708    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zpt\" (UniqueName: \"kubernetes.io/projected/7a1df436-5c89-4c81-b77a-61580da3b564-kube-api-access-m8zpt\") pod \"maas-api-revoke-sub-f065f6cd-59qhv\" (UID: \"7a1df436-5c89-4c81-b77a-61580da3b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:40.177604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:40.170673    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f065f6cd-59qhv\" (UID: \"7a1df436-5c89-4c81-b77a-61580da3b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:40.211086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:40.211050    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f065f6cd-59qhv\" (UID: \"7a1df436-5c89-4c81-b77a-61580da3b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:40.224610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:40.223158    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zpt\" (UniqueName: \"kubernetes.io/projected/7a1df436-5c89-4c81-b77a-61580da3b564-kube-api-access-m8zpt\") pod \"maas-api-revoke-sub-f065f6cd-59qhv\" (UID: \"7a1df436-5c89-4c81-b77a-61580da3b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:40.414053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:40.413729    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:41.962637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:41.962503    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:09:41.971605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:41.971579    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"]
Oct 30 14:09:42.116524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:42.116385    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv" event={"ID":"7a1df436-5c89-4c81-b77a-61580da3b564","Type":"ContainerStarted","Data":"3c7c70bd6c072823e3472cf360c5d5e200dca120a48339c490221f215d88e1ca"}
Oct 30 14:09:42.117865 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:09:42.117837    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:09:43.125777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:43.125743    4530 generic.go:356] "Generic (PLEG): container finished" podID="7a1df436-5c89-4c81-b77a-61580da3b564" containerID="98cbb65a76be90d7cb7ed3010b82d109924e34f55fce20d34300dfd0f6f1f7e6" exitCode=0
Oct 30 14:09:43.126128 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:43.125831    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv" event={"ID":"7a1df436-5c89-4c81-b77a-61580da3b564","Type":"ContainerDied","Data":"98cbb65a76be90d7cb7ed3010b82d109924e34f55fce20d34300dfd0f6f1f7e6"}
Oct 30 14:09:44.282578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:44.282526    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:44.370393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:44.370357    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-xks-ca\") pod \"7a1df436-5c89-4c81-b77a-61580da3b564\" (UID: \"7a1df436-5c89-4c81-b77a-61580da3b564\") "
Oct 30 14:09:44.370594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:44.370429    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-openshift-ca\") pod \"7a1df436-5c89-4c81-b77a-61580da3b564\" (UID: \"7a1df436-5c89-4c81-b77a-61580da3b564\") "
Oct 30 14:09:44.370674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:44.370610    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a1df436-5c89-4c81-b77a-61580da3b564-kube-api-access-m8zpt\" (UniqueName: \"kubernetes.io/projected/7a1df436-5c89-4c81-b77a-61580da3b564-kube-api-access-m8zpt\") pod \"7a1df436-5c89-4c81-b77a-61580da3b564\" (UID: \"7a1df436-5c89-4c81-b77a-61580da3b564\") "
Oct 30 14:09:44.370885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:44.370854    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-openshift-ca" pod "7a1df436-5c89-4c81-b77a-61580da3b564" (UID: "7a1df436-5c89-4c81-b77a-61580da3b564"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:09:44.371362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:44.371336    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:44.372456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:44.372434    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-xks-ca" pod "7a1df436-5c89-4c81-b77a-61580da3b564" (UID: "7a1df436-5c89-4c81-b77a-61580da3b564"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:44.372657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:44.372641    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1df436-5c89-4c81-b77a-61580da3b564-kube-api-access-m8zpt" pod "7a1df436-5c89-4c81-b77a-61580da3b564" (UID: "7a1df436-5c89-4c81-b77a-61580da3b564"). InnerVolumeSpecName "kube-api-access-m8zpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:44.472575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:44.472501    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1df436-5c89-4c81-b77a-61580da3b564-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:44.472575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:44.472528    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8zpt\" (UniqueName: \"kubernetes.io/projected/7a1df436-5c89-4c81-b77a-61580da3b564-kube-api-access-m8zpt\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:45.142162 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:45.142130    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:09:45.142300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:45.142168    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv" event={"ID":"7a1df436-5c89-4c81-b77a-61580da3b564","Type":"ContainerDied","Data":"3c7c70bd6c072823e3472cf360c5d5e200dca120a48339c490221f215d88e1ca"}
Oct 30 14:09:45.142300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:45.142214    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c7c70bd6c072823e3472cf360c5d5e200dca120a48339c490221f215d88e1ca"
Oct 30 14:09:50.168702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.168666    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"]
Oct 30 14:09:50.169422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.169401    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7a1df436-5c89-4c81-b77a-61580da3b564" containerName="revoke-keys"
Oct 30 14:09:50.169422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.169423    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1df436-5c89-4c81-b77a-61580da3b564" containerName="revoke-keys"
Oct 30 14:09:50.169564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.169552    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a1df436-5c89-4c81-b77a-61580da3b564" containerName="revoke-keys"
Oct 30 14:09:50.173304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.173287    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:50.193825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.193799    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"]
Oct 30 14:09:50.200252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.200226    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:09:50.200370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.200316    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:09:50.284319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.284283    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-557ad36c-sdc8r\" (UID: \"a9681345-6f5c-4527-9b68-e86a1f66b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:50.284458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.284444    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-557ad36c-sdc8r\" (UID: \"a9681345-6f5c-4527-9b68-e86a1f66b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:50.284606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.284563    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnlcb\" (UniqueName: \"kubernetes.io/projected/a9681345-6f5c-4527-9b68-e86a1f66b15d-kube-api-access-bnlcb\") pod \"maas-api-revoke-sub-557ad36c-sdc8r\" (UID: \"a9681345-6f5c-4527-9b68-e86a1f66b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:50.385385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.385349    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-557ad36c-sdc8r\" (UID: \"a9681345-6f5c-4527-9b68-e86a1f66b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:50.385583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.385421    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnlcb\" (UniqueName: \"kubernetes.io/projected/a9681345-6f5c-4527-9b68-e86a1f66b15d-kube-api-access-bnlcb\") pod \"maas-api-revoke-sub-557ad36c-sdc8r\" (UID: \"a9681345-6f5c-4527-9b68-e86a1f66b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:50.385583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.385501    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-557ad36c-sdc8r\" (UID: \"a9681345-6f5c-4527-9b68-e86a1f66b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:50.386079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.386052    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-557ad36c-sdc8r\" (UID: \"a9681345-6f5c-4527-9b68-e86a1f66b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:50.387869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.387847    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-557ad36c-sdc8r\" (UID: \"a9681345-6f5c-4527-9b68-e86a1f66b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:50.411830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.411801    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnlcb\" (UniqueName: \"kubernetes.io/projected/a9681345-6f5c-4527-9b68-e86a1f66b15d-kube-api-access-bnlcb\") pod \"maas-api-revoke-sub-557ad36c-sdc8r\" (UID: \"a9681345-6f5c-4527-9b68-e86a1f66b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:50.488212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.488154    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:50.796778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:50.796746    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"]
Oct 30 14:09:51.263351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:51.263316    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r" event={"ID":"a9681345-6f5c-4527-9b68-e86a1f66b15d","Type":"ContainerStarted","Data":"6a95e04df92e872810215699332ad2c63890104525e62c84ffd484dd6a10a1d3"}
Oct 30 14:09:51.263714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:51.263370    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r" event={"ID":"a9681345-6f5c-4527-9b68-e86a1f66b15d","Type":"ContainerStarted","Data":"124f6d7c596a056d67daf17b38b25ffca68e922b5f537995941e8597a3c1ecdc"}
Oct 30 14:09:51.310890 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:51.310847    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r" podStartSLOduration=1.310833957 podStartE2EDuration="1.310833957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:51.308598796 +0000 UTC m=+1256.340497736" watchObservedRunningTime="2026-10-30 14:09:51.310833957 +0000 UTC m=+1256.342732918"
Oct 30 14:09:52.024821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:52.024742    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:09:52.272238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:52.272200    4530 generic.go:356] "Generic (PLEG): container finished" podID="a9681345-6f5c-4527-9b68-e86a1f66b15d" containerID="6a95e04df92e872810215699332ad2c63890104525e62c84ffd484dd6a10a1d3" exitCode=0
Oct 30 14:09:52.272641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:52.272271    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r" event={"ID":"a9681345-6f5c-4527-9b68-e86a1f66b15d","Type":"ContainerDied","Data":"6a95e04df92e872810215699332ad2c63890104525e62c84ffd484dd6a10a1d3"}
Oct 30 14:09:54.947727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:54.947686    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"]
Oct 30 14:09:55.017758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.016621    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.218502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.186044    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b0d295-010d-4193-8d2d-c9dc02150bd8-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.218991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.218955    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.219206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.219179    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.239852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.219497    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlq5\" (UniqueName: \"kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-kube-api-access-bzlq5\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.240440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.240404    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.240668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.240647    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.240828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.240810    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.280566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.261125    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.280566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.261428    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.361566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.347196    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"]
Oct 30 14:09:55.372572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.369930    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:55.372572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.372261    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.372572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.372311    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.372572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.372354    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.372572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.372410    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.372572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.372482    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b0d295-010d-4193-8d2d-c9dc02150bd8-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.373927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.372531    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.399994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.399959    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b0d295-010d-4193-8d2d-c9dc02150bd8-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.469612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.373902    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.469612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.451977    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlq5\" (UniqueName: \"kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-kube-api-access-bzlq5\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.469612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.452208    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.469612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.466610    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.469612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.466817    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.505103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.505052    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.506461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.506437    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.627564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.618959    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.728340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.728251    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-xks-ca\") pod \"a9681345-6f5c-4527-9b68-e86a1f66b15d\" (UID: \"a9681345-6f5c-4527-9b68-e86a1f66b15d\") "
Oct 30 14:09:55.778577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.728928    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9681345-6f5c-4527-9b68-e86a1f66b15d-kube-api-access-bnlcb\" (UniqueName: \"kubernetes.io/projected/a9681345-6f5c-4527-9b68-e86a1f66b15d-kube-api-access-bnlcb\") pod \"a9681345-6f5c-4527-9b68-e86a1f66b15d\" (UID: \"a9681345-6f5c-4527-9b68-e86a1f66b15d\") "
Oct 30 14:09:55.778577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.756762    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-openshift-ca\") pod \"a9681345-6f5c-4527-9b68-e86a1f66b15d\" (UID: \"a9681345-6f5c-4527-9b68-e86a1f66b15d\") "
Oct 30 14:09:55.778577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.771847    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:09:55.895567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.805553    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-xks-ca" pod "a9681345-6f5c-4527-9b68-e86a1f66b15d" (UID: "a9681345-6f5c-4527-9b68-e86a1f66b15d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:55.895567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.838776    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9681345-6f5c-4527-9b68-e86a1f66b15d-kube-api-access-bnlcb" pod "a9681345-6f5c-4527-9b68-e86a1f66b15d" (UID: "a9681345-6f5c-4527-9b68-e86a1f66b15d"). InnerVolumeSpecName "kube-api-access-bnlcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:55.907886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.907831    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-openshift-ca" pod "a9681345-6f5c-4527-9b68-e86a1f66b15d" (UID: "a9681345-6f5c-4527-9b68-e86a1f66b15d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:09:55.966564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.955568    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:55.966564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:55.956106    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:56.232831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:56.193332    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnlcb\" (UniqueName: \"kubernetes.io/projected/a9681345-6f5c-4527-9b68-e86a1f66b15d-kube-api-access-bnlcb\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:56.232831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:56.193378    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:56.232831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:56.193396    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9681345-6f5c-4527-9b68-e86a1f66b15d-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:09:56.335104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:56.335065    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlq5\" (UniqueName: \"kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-kube-api-access-bzlq5\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:56.510754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:56.353303    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:09:56.762794 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:56.762710    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:09:57.096582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:57.076840    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r" event={"ID":"a9681345-6f5c-4527-9b68-e86a1f66b15d","Type":"ContainerDied","Data":"124f6d7c596a056d67daf17b38b25ffca68e922b5f537995941e8597a3c1ecdc"}
Oct 30 14:09:57.096582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:57.076919    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="124f6d7c596a056d67daf17b38b25ffca68e922b5f537995941e8597a3c1ecdc"
Oct 30 14:09:57.096582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:57.076992    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"
Oct 30 14:09:58.653566 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:09:58.650206    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b0d295_010d_4193_8d2d_c9dc02150bd8.slice/crio-f1615640ba755b6a0a5b2102ded9cb3ce3a54d53c69f3a3dcd2df30386b4f2f4 WatchSource:0}: Error finding container f1615640ba755b6a0a5b2102ded9cb3ce3a54d53c69f3a3dcd2df30386b4f2f4: Status 404 returned error can't find the container with id f1615640ba755b6a0a5b2102ded9cb3ce3a54d53c69f3a3dcd2df30386b4f2f4
Oct 30 14:09:58.693716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:58.689270    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:09:58.693716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:58.689397    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:09:58.693716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:58.689446    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:09:58.712570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:58.707306    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"]
Oct 30 14:09:59.306315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:59.306270    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq" event={"ID":"59b0d295-010d-4193-8d2d-c9dc02150bd8","Type":"ContainerStarted","Data":"a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09"}
Oct 30 14:09:59.306629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:59.306334    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq" event={"ID":"59b0d295-010d-4193-8d2d-c9dc02150bd8","Type":"ContainerStarted","Data":"f1615640ba755b6a0a5b2102ded9cb3ce3a54d53c69f3a3dcd2df30386b4f2f4"}
Oct 30 14:09:59.360500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:59.360449    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq" podStartSLOduration=5.360434925 podStartE2EDuration="5.360434925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:59.356597944 +0000 UTC m=+1264.388496897" watchObservedRunningTime="2026-10-30 14:09:59.360434925 +0000 UTC m=+1264.392333864"
Oct 30 14:09:59.764005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:09:59.763930    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:10:00.789567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:00.786095    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 14:10:00.789567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:00.786176    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq" podUID="59b0d295-010d-4193-8d2d-c9dc02150bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 14:10:01.764296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:01.764260    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 14:10:01.764461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:01.764310    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq" podUID="59b0d295-010d-4193-8d2d-c9dc02150bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 14:10:02.763680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:02.763643    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 14:10:02.764093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:02.763702    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq" podUID="59b0d295-010d-4193-8d2d-c9dc02150bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 14:10:03.764106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:03.764071    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 14:10:03.764493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:03.764123    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq" podUID="59b0d295-010d-4193-8d2d-c9dc02150bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 14:10:04.763825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:04.763786    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 14:10:04.764006 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:04.763840    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq" podUID="59b0d295-010d-4193-8d2d-c9dc02150bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 14:10:06.097679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:06.097648    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:10:06.098524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:06.098469    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:10:06.099427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:06.099406    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:10:09.785118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.785085    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"]
Oct 30 14:10:09.785900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.785877    4530 cpu_manager.go:414] "containerMap: removing container" podUID="a9681345-6f5c-4527-9b68-e86a1f66b15d" containerName="revoke-keys"
Oct 30 14:10:09.785900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.785897    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9681345-6f5c-4527-9b68-e86a1f66b15d" containerName="revoke-keys"
Oct 30 14:10:09.786116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.786060    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9681345-6f5c-4527-9b68-e86a1f66b15d" containerName="revoke-keys"
Oct 30 14:10:09.789820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.789795    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:09.815530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.815494    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"]
Oct 30 14:10:09.966505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.966469    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:09.966687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.966525    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:09.966734 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.966680    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:09.966770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.966734    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:09.966770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.966757    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54eb3905-9d69-4cf9-8cc8-a279635e8183-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:09.966830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.966779    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6mwr\" (UniqueName: \"kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-kube-api-access-v6mwr\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:09.966889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.966876    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:09.966938 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.966927    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:09.967040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:09.967025    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.068665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.068637    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.068812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.068690    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.068812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.068741    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.068812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.068766    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.068812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.068800    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.069005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.068921    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.069005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.068965    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54eb3905-9d69-4cf9-8cc8-a279635e8183-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.069005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.068997    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6mwr\" (UniqueName: \"kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-kube-api-access-v6mwr\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.069204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.069083    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.069204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.069106    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.069204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.069150    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.069401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.069298    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.069401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.069340    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.069901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.069877    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54eb3905-9d69-4cf9-8cc8-a279635e8183-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.071143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.071120    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.071418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.071398    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.095575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.095521    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.095750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.095731    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6mwr\" (UniqueName: \"kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-kube-api-access-v6mwr\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.114660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.114639    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:10.440200 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:10:10.440172    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54eb3905_9d69_4cf9_8cc8_a279635e8183.slice/crio-5a8d397cb672cec3e2a2becc2efe1e234341d74fb49bc15d659537b211dd2332 WatchSource:0}: Error finding container 5a8d397cb672cec3e2a2becc2efe1e234341d74fb49bc15d659537b211dd2332: Status 404 returned error can't find the container with id 5a8d397cb672cec3e2a2becc2efe1e234341d74fb49bc15d659537b211dd2332
Oct 30 14:10:10.443224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.443185    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:10:10.443339 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.443267    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:10:10.443339 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.443311    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:10:10.443968 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.443946    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"]
Oct 30 14:10:10.471278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.471254    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" event={"ID":"54eb3905-9d69-4cf9-8cc8-a279635e8183","Type":"ContainerStarted","Data":"5a8d397cb672cec3e2a2becc2efe1e234341d74fb49bc15d659537b211dd2332"}
Oct 30 14:10:10.549777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.549736    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"]
Oct 30 14:10:10.555742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.555714    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.588750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.587454    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"]
Oct 30 14:10:10.677726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.677689    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.677898 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.677840    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.677898 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.677876    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.677898 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.677894    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zmd8\" (UniqueName: \"kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-kube-api-access-2zmd8\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.678011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.677989    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.678048 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.678036    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.678148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.678136    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.678207 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.678196    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.678298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.678285    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.779606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.779460    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.779891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.779844    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.779891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.779889    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmd8\" (UniqueName: \"kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-kube-api-access-2zmd8\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.780033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.779980    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.780075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.780052    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.780168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.780150    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.780406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.780233    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.780485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.780387    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.780485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.780442    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.780485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.780422    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.780485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.780469    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.780745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.780513    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.780800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.780765    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.781510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.781485    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.782707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.782682    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.783062 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.783041    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.807985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.807960    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.808386 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.808046    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmd8\" (UniqueName: \"kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-kube-api-access-2zmd8\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:10.880316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:10.880284    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:11.207886 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:10:11.207851    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18ce5d6_43d9_4210_a54d_329f8ae1e3d5.slice/crio-80e83622c800a63e95ff53ddee72b00c9c7f9321c63b4ad9d722c8fd7702a0ac WatchSource:0}: Error finding container 80e83622c800a63e95ff53ddee72b00c9c7f9321c63b4ad9d722c8fd7702a0ac: Status 404 returned error can't find the container with id 80e83622c800a63e95ff53ddee72b00c9c7f9321c63b4ad9d722c8fd7702a0ac
Oct 30 14:10:11.210700 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.210662    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:10:11.210795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.210752    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:10:11.210795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.210784    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:10:11.211383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.211359    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"]
Oct 30 14:10:11.371556 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:10:11.371507    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:10:11.519142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.519061    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" event={"ID":"54eb3905-9d69-4cf9-8cc8-a279635e8183","Type":"ContainerStarted","Data":"b8a69ae8796b70107b66408024a1ce4e2099ac2d2b6b14ff987fd763214f1a34"}
Oct 30 14:10:11.521370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.521339    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" event={"ID":"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5","Type":"ContainerStarted","Data":"7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03"}
Oct 30 14:10:11.521494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.521394    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" event={"ID":"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5","Type":"ContainerStarted","Data":"80e83622c800a63e95ff53ddee72b00c9c7f9321c63b4ad9d722c8fd7702a0ac"}
Oct 30 14:10:11.569421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.569376    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" podStartSLOduration=2.569362124 podStartE2EDuration="2.569362124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:11.568676304 +0000 UTC m=+1276.600575243" watchObservedRunningTime="2026-10-30 14:10:11.569362124 +0000 UTC m=+1276.601261063"
Oct 30 14:10:11.645383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.645314    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podStartSLOduration=1.645293256 podStartE2EDuration="1.645293256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:11.619171912 +0000 UTC m=+1276.651070852" watchObservedRunningTime="2026-10-30 14:10:11.645293256 +0000 UTC m=+1276.677192196"
Oct 30 14:10:11.852673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.852644    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"]
Oct 30 14:10:11.858785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.858764    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"]
Oct 30 14:10:11.858973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.858950    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:11.877829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.877804    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:10:11.877994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.877978    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:10:11.893567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.887623    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:11.945773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.945739    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48391366-4znkv\" (UID: \"ac9d860c-4982-42b2-bea5-418b31649631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:11.945932 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.945859    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxhf\" (UniqueName: \"kubernetes.io/projected/ac9d860c-4982-42b2-bea5-418b31649631-kube-api-access-bfxhf\") pod \"maas-api-revoke-sub-48391366-4znkv\" (UID: \"ac9d860c-4982-42b2-bea5-418b31649631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:11.946026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:11.946007    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48391366-4znkv\" (UID: \"ac9d860c-4982-42b2-bea5-418b31649631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:12.047246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.047213    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxhf\" (UniqueName: \"kubernetes.io/projected/ac9d860c-4982-42b2-bea5-418b31649631-kube-api-access-bfxhf\") pod \"maas-api-revoke-sub-48391366-4znkv\" (UID: \"ac9d860c-4982-42b2-bea5-418b31649631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:12.047401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.047332    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48391366-4znkv\" (UID: \"ac9d860c-4982-42b2-bea5-418b31649631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:12.047453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.047432    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48391366-4znkv\" (UID: \"ac9d860c-4982-42b2-bea5-418b31649631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:12.048122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.048096    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48391366-4znkv\" (UID: \"ac9d860c-4982-42b2-bea5-418b31649631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:12.049591 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.049569    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48391366-4znkv\" (UID: \"ac9d860c-4982-42b2-bea5-418b31649631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:12.073888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.073863    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxhf\" (UniqueName: \"kubernetes.io/projected/ac9d860c-4982-42b2-bea5-418b31649631-kube-api-access-bfxhf\") pod \"maas-api-revoke-sub-48391366-4znkv\" (UID: \"ac9d860c-4982-42b2-bea5-418b31649631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:12.115000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.114920    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:12.117065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.117031    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 14:10:12.117226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.117080    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 14:10:12.204629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.203629    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:12.537603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.537569    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"]
Oct 30 14:10:12.913565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.909687    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 14:10:12.913565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:12.909790    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 14:10:13.119869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:13.119351    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 14:10:13.119869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:13.119421    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 14:10:13.561598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:13.561557    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv" event={"ID":"ac9d860c-4982-42b2-bea5-418b31649631","Type":"ContainerStarted","Data":"2756000688f45d5fb81e28201007da27f0f6cc6eaabb1f6694bbe2664cd21aea"}
Oct 30 14:10:13.561776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:13.561621    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv" event={"ID":"ac9d860c-4982-42b2-bea5-418b31649631","Type":"ContainerStarted","Data":"6680ad539c3606f7f4e97ed7e3817d229a5dc761f51c7d73a6cbf91975dc1974"}
Oct 30 14:10:13.883940 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:13.883851    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 14:10:13.884192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:13.884165    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 14:10:14.121565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:14.120004    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 14:10:14.121565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:14.120074    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 14:10:14.634738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:14.634699    4530 generic.go:356] "Generic (PLEG): container finished" podID="ac9d860c-4982-42b2-bea5-418b31649631" containerID="2756000688f45d5fb81e28201007da27f0f6cc6eaabb1f6694bbe2664cd21aea" exitCode=0
Oct 30 14:10:14.635356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:14.635326    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv" event={"ID":"ac9d860c-4982-42b2-bea5-418b31649631","Type":"ContainerDied","Data":"2756000688f45d5fb81e28201007da27f0f6cc6eaabb1f6694bbe2664cd21aea"}
Oct 30 14:10:14.903621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:14.887926    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 14:10:14.903621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:14.887985    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 14:10:15.124131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:15.124094    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 14:10:15.124745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:15.124714    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 14:10:15.921618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:15.894948    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 14:10:15.921618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:15.895007    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 14:10:16.076750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.076727    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:16.139944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.135924    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 14:10:16.139944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.136011    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 14:10:16.164386 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.164346    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9d860c-4982-42b2-bea5-418b31649631-kube-api-access-bfxhf\" (UniqueName: \"kubernetes.io/projected/ac9d860c-4982-42b2-bea5-418b31649631-kube-api-access-bfxhf\") pod \"ac9d860c-4982-42b2-bea5-418b31649631\" (UID: \"ac9d860c-4982-42b2-bea5-418b31649631\") "
Oct 30 14:10:16.164581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.164408    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-xks-ca\") pod \"ac9d860c-4982-42b2-bea5-418b31649631\" (UID: \"ac9d860c-4982-42b2-bea5-418b31649631\") "
Oct 30 14:10:16.164673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.164596    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-openshift-ca\") pod \"ac9d860c-4982-42b2-bea5-418b31649631\" (UID: \"ac9d860c-4982-42b2-bea5-418b31649631\") "
Oct 30 14:10:16.165160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.165124    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-openshift-ca" pod "ac9d860c-4982-42b2-bea5-418b31649631" (UID: "ac9d860c-4982-42b2-bea5-418b31649631"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:16.165436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.165413    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:16.167361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.167336    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-xks-ca" pod "ac9d860c-4982-42b2-bea5-418b31649631" (UID: "ac9d860c-4982-42b2-bea5-418b31649631"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:16.169227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.169168    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9d860c-4982-42b2-bea5-418b31649631-kube-api-access-bfxhf" pod "ac9d860c-4982-42b2-bea5-418b31649631" (UID: "ac9d860c-4982-42b2-bea5-418b31649631"). InnerVolumeSpecName "kube-api-access-bfxhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:16.267197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.267168    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfxhf\" (UniqueName: \"kubernetes.io/projected/ac9d860c-4982-42b2-bea5-418b31649631-kube-api-access-bfxhf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:16.267399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.267382    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9d860c-4982-42b2-bea5-418b31649631-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:16.740371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.740319    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv" event={"ID":"ac9d860c-4982-42b2-bea5-418b31649631","Type":"ContainerDied","Data":"6680ad539c3606f7f4e97ed7e3817d229a5dc761f51c7d73a6cbf91975dc1974"}
Oct 30 14:10:16.740681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.740645    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6680ad539c3606f7f4e97ed7e3817d229a5dc761f51c7d73a6cbf91975dc1974"
Oct 30 14:10:16.740905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.740892    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"
Oct 30 14:10:16.931567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.929090    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 14:10:16.931567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:16.929168    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 14:10:17.373564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:17.314810    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 14:10:17.373564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:17.314881    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 14:10:18.223092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:18.223052    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 14:10:18.264796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:18.264737    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 14:10:18.291323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:18.291283    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 14:10:18.316567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:18.291581    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 14:10:18.383141 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:10:18.381603    4530 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.223s"
Oct 30 14:10:18.762571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:18.755088    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:10:18.881757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:18.881720    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 14:10:18.882040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:18.882000    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 14:10:19.121177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:19.121146    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 14:10:19.121928 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:19.121442    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 14:10:19.843627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:19.840680    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:10:19.843627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:19.841776    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:10:19.891219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:19.884468    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 14:10:19.891219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:19.884532    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 14:10:20.128779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:20.128666    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 14:10:20.150725 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:20.150682    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 14:10:20.399087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:20.398988    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:10:20.903566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:20.886978    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 14:10:20.903566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:20.887037    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 14:10:21.882564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:21.878447    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:21.882564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:21.880223    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:21.882564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:21.881334    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:10:22.278074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:22.277991    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:22.278533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:22.278431    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:22.279429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:22.279403    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:10:24.969790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:24.969758    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"]
Oct 30 14:10:24.970413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:24.970397    4530 cpu_manager.go:414] "containerMap: removing container" podUID="ac9d860c-4982-42b2-bea5-418b31649631" containerName="revoke-keys"
Oct 30 14:10:24.970413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:24.970413    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9d860c-4982-42b2-bea5-418b31649631" containerName="revoke-keys"
Oct 30 14:10:24.970535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:24.970526    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9d860c-4982-42b2-bea5-418b31649631" containerName="revoke-keys"
Oct 30 14:10:24.974669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:24.974646    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:24.993129 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:24.993102    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"]
Oct 30 14:10:25.000900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.000869    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:10:25.001327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.001301    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:10:25.002726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.002697    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4414981d-h9ngx\" (UID: \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:25.002888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.002803    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4414981d-h9ngx\" (UID: \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:25.003016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.002998    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rc68\" (UniqueName: \"kubernetes.io/projected/8672c30c-ad28-4aff-bda5-1adbcf8dd456-kube-api-access-5rc68\") pod \"maas-api-revoke-sub-4414981d-h9ngx\" (UID: \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:25.105210 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.105176    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4414981d-h9ngx\" (UID: \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:25.105461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.105438    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4414981d-h9ngx\" (UID: \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:25.105729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.105708    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rc68\" (UniqueName: \"kubernetes.io/projected/8672c30c-ad28-4aff-bda5-1adbcf8dd456-kube-api-access-5rc68\") pod \"maas-api-revoke-sub-4414981d-h9ngx\" (UID: \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:25.107366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.107344    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4414981d-h9ngx\" (UID: \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:25.117853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.117829    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4414981d-h9ngx\" (UID: \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:25.132885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.132857    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rc68\" (UniqueName: \"kubernetes.io/projected/8672c30c-ad28-4aff-bda5-1adbcf8dd456-kube-api-access-5rc68\") pod \"maas-api-revoke-sub-4414981d-h9ngx\" (UID: \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:25.289710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.289658    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:25.595082 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:10:25.595046    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8672c30c_ad28_4aff_bda5_1adbcf8dd456.slice/crio-8ef45b81330dc478b059f44509f75320c83351d7fc703b2f3ac894f2678c880b WatchSource:0}: Error finding container 8ef45b81330dc478b059f44509f75320c83351d7fc703b2f3ac894f2678c880b: Status 404 returned error can't find the container with id 8ef45b81330dc478b059f44509f75320c83351d7fc703b2f3ac894f2678c880b
Oct 30 14:10:25.598132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.598107    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"]
Oct 30 14:10:25.701507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:25.701476    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx" event={"ID":"8672c30c-ad28-4aff-bda5-1adbcf8dd456","Type":"ContainerStarted","Data":"8ef45b81330dc478b059f44509f75320c83351d7fc703b2f3ac894f2678c880b"}
Oct 30 14:10:26.710688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.710653    4530 generic.go:356] "Generic (PLEG): container finished" podID="8672c30c-ad28-4aff-bda5-1adbcf8dd456" containerID="fe2afd4906a470e8e6f3ffdc81ae66b793f60159e8b35ed3c961db2f9563e7c6" exitCode=0
Oct 30 14:10:26.711104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.710738    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx" event={"ID":"8672c30c-ad28-4aff-bda5-1adbcf8dd456","Type":"ContainerDied","Data":"fe2afd4906a470e8e6f3ffdc81ae66b793f60159e8b35ed3c961db2f9563e7c6"}
Oct 30 14:10:26.751549 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.751438    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"]
Oct 30 14:10:26.757090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.757051    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.777555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.777188    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"distinct-2-w1-ca229c-kserve-self-signed-certs\""
Oct 30 14:10:26.777555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.777186    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"kube-root-ca.crt\""
Oct 30 14:10:26.777555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.777254    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"default-dockercfg-499ms\""
Oct 30 14:10:26.777555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.777326    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"openshift-service-ca.crt\""
Oct 30 14:10:26.777860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.777723    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"]
Oct 30 14:10:26.854276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.854241    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tls-certs\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.854415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.854363    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5vdm\" (UniqueName: \"kubernetes.io/projected/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-kube-api-access-n5vdm\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.854486 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.854470    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tmp-dir\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.854626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.854610    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-dshm\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.854669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.854652    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-model-cache\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.854791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.854779    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-home\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.960565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.960014    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-home\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.960565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.960146    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tls-certs\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.960565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.960214    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5vdm\" (UniqueName: \"kubernetes.io/projected/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-kube-api-access-n5vdm\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.960565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.960318    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tmp-dir\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.960565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.960415    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-dshm\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.960565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.960453    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-model-cache\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.961037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.960951    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-model-cache\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.979570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.961240    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-home\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.979570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.961997    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tmp-dir\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.997562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.989578    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-dshm\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:26.997562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:26.991640    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5vdm\" (UniqueName: \"kubernetes.io/projected/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-kube-api-access-n5vdm\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:27.016149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:27.016072    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tls-certs\") pod \"distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:27.085906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:27.085870    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:27.496076 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:10:27.496037    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10b2c1ea_30bd_44be_baaf_5fe82ee558a5.slice/crio-ea282c0f85f0e7eeb446a341e870a9dc27202e3aa6d35edda0cc2d792a05e6de WatchSource:0}: Error finding container ea282c0f85f0e7eeb446a341e870a9dc27202e3aa6d35edda0cc2d792a05e6de: Status 404 returned error can't find the container with id ea282c0f85f0e7eeb446a341e870a9dc27202e3aa6d35edda0cc2d792a05e6de
Oct 30 14:10:27.498809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:27.498783    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"]
Oct 30 14:10:27.751768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:27.751676    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" event={"ID":"10b2c1ea-30bd-44be-baaf-5fe82ee558a5","Type":"ContainerStarted","Data":"6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b"}
Oct 30 14:10:27.752352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:27.752316    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" event={"ID":"10b2c1ea-30bd-44be-baaf-5fe82ee558a5","Type":"ContainerStarted","Data":"ea282c0f85f0e7eeb446a341e870a9dc27202e3aa6d35edda0cc2d792a05e6de"}
Oct 30 14:10:27.813440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:27.812310    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podStartSLOduration=1.812290275 podStartE2EDuration="1.812290275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:27.804231072 +0000 UTC m=+1292.836130011" watchObservedRunningTime="2026-10-30 14:10:27.812290275 +0000 UTC m=+1292.844189216"
Oct 30 14:10:27.912749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:27.912724    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:27.998588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:27.996492    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8672c30c-ad28-4aff-bda5-1adbcf8dd456-kube-api-access-5rc68\" (UniqueName: \"kubernetes.io/projected/8672c30c-ad28-4aff-bda5-1adbcf8dd456-kube-api-access-5rc68\") pod \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\" (UID: \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\") "
Oct 30 14:10:27.998588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:27.996614    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-xks-ca\") pod \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\" (UID: \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\") "
Oct 30 14:10:27.998588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:27.996788    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-openshift-ca\") pod \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\" (UID: \"8672c30c-ad28-4aff-bda5-1adbcf8dd456\") "
Oct 30 14:10:28.012429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:28.012321    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8672c30c-ad28-4aff-bda5-1adbcf8dd456-kube-api-access-5rc68" pod "8672c30c-ad28-4aff-bda5-1adbcf8dd456" (UID: "8672c30c-ad28-4aff-bda5-1adbcf8dd456"). InnerVolumeSpecName "kube-api-access-5rc68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:28.037558 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:28.032705    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-xks-ca" pod "8672c30c-ad28-4aff-bda5-1adbcf8dd456" (UID: "8672c30c-ad28-4aff-bda5-1adbcf8dd456"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:28.055603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:28.054869    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-openshift-ca" pod "8672c30c-ad28-4aff-bda5-1adbcf8dd456" (UID: "8672c30c-ad28-4aff-bda5-1adbcf8dd456"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:28.087377 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:28.087346    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:28.099371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:28.099337    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rc68\" (UniqueName: \"kubernetes.io/projected/8672c30c-ad28-4aff-bda5-1adbcf8dd456-kube-api-access-5rc68\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:28.099371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:28.099364    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:28.099532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:28.099379    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8672c30c-ad28-4aff-bda5-1adbcf8dd456-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:28.760206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:28.760174    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"
Oct 30 14:10:28.760611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:28.760207    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx" event={"ID":"8672c30c-ad28-4aff-bda5-1adbcf8dd456","Type":"ContainerDied","Data":"8ef45b81330dc478b059f44509f75320c83351d7fc703b2f3ac894f2678c880b"}
Oct 30 14:10:28.760611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:28.760256    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ef45b81330dc478b059f44509f75320c83351d7fc703b2f3ac894f2678c880b"
Oct 30 14:10:34.386442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:34.386397    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:10:36.135567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.133436    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"]
Oct 30 14:10:36.135567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.134661    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8672c30c-ad28-4aff-bda5-1adbcf8dd456" containerName="revoke-keys"
Oct 30 14:10:36.135567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.134682    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8672c30c-ad28-4aff-bda5-1adbcf8dd456" containerName="revoke-keys"
Oct 30 14:10:36.135567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.134856    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8672c30c-ad28-4aff-bda5-1adbcf8dd456" containerName="revoke-keys"
Oct 30 14:10:36.145565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.141951    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:36.174602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.167610    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:10:36.182581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.181638    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:10:36.189920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.189890    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mtv8\" (UniqueName: \"kubernetes.io/projected/ec4079be-b302-497b-9e34-abe0e7208507-kube-api-access-5mtv8\") pod \"maas-api-revoke-sub-fd5beda1-wthzr\" (UID: \"ec4079be-b302-497b-9e34-abe0e7208507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:36.190114 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.190094    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd5beda1-wthzr\" (UID: \"ec4079be-b302-497b-9e34-abe0e7208507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:36.190829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.190805    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd5beda1-wthzr\" (UID: \"ec4079be-b302-497b-9e34-abe0e7208507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:36.212685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.212593    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"]
Oct 30 14:10:36.303973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.303939    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd5beda1-wthzr\" (UID: \"ec4079be-b302-497b-9e34-abe0e7208507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:36.317485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.317438    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mtv8\" (UniqueName: \"kubernetes.io/projected/ec4079be-b302-497b-9e34-abe0e7208507-kube-api-access-5mtv8\") pod \"maas-api-revoke-sub-fd5beda1-wthzr\" (UID: \"ec4079be-b302-497b-9e34-abe0e7208507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:36.317999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.317978    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd5beda1-wthzr\" (UID: \"ec4079be-b302-497b-9e34-abe0e7208507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:36.319117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.319095    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd5beda1-wthzr\" (UID: \"ec4079be-b302-497b-9e34-abe0e7208507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:36.371066 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.371040    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd5beda1-wthzr\" (UID: \"ec4079be-b302-497b-9e34-abe0e7208507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:36.430307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.430224    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mtv8\" (UniqueName: \"kubernetes.io/projected/ec4079be-b302-497b-9e34-abe0e7208507-kube-api-access-5mtv8\") pod \"maas-api-revoke-sub-fd5beda1-wthzr\" (UID: \"ec4079be-b302-497b-9e34-abe0e7208507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:36.486834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:36.486796    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:37.106567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:37.102898    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:38.134568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:38.125383    4530 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": context deadline exceeded"
Oct 30 14:10:38.518728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:38.518623    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Oct 30 14:10:38.572887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:38.572844    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:10:38.573159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:38.573130    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:10:38.632507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:38.632468    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr" event={"ID":"ec4079be-b302-497b-9e34-abe0e7208507","Type":"ContainerStarted","Data":"0f5e5a93ab7baa4a9fd3df65de7273a99cd1e7023d961d046f7d2c4983b191c6"}
Oct 30 14:10:38.662564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:38.658902    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"]
Oct 30 14:10:39.642061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:39.642033    4530 generic.go:356] "Generic (PLEG): container finished" podID="ec4079be-b302-497b-9e34-abe0e7208507" containerID="97c5001a9b48bc14308076bdc284375e836957350c84d6a763738f7a28a3baf6" exitCode=0
Oct 30 14:10:39.642395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:39.642113    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr" event={"ID":"ec4079be-b302-497b-9e34-abe0e7208507","Type":"ContainerDied","Data":"97c5001a9b48bc14308076bdc284375e836957350c84d6a763738f7a28a3baf6"}
Oct 30 14:10:40.812388 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:40.812360    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:40.874022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:40.873990    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-openshift-ca\") pod \"ec4079be-b302-497b-9e34-abe0e7208507\" (UID: \"ec4079be-b302-497b-9e34-abe0e7208507\") "
Oct 30 14:10:40.874197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:40.874182    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4079be-b302-497b-9e34-abe0e7208507-kube-api-access-5mtv8\" (UniqueName: \"kubernetes.io/projected/ec4079be-b302-497b-9e34-abe0e7208507-kube-api-access-5mtv8\") pod \"ec4079be-b302-497b-9e34-abe0e7208507\" (UID: \"ec4079be-b302-497b-9e34-abe0e7208507\") "
Oct 30 14:10:40.874280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:40.874244    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-xks-ca\") pod \"ec4079be-b302-497b-9e34-abe0e7208507\" (UID: \"ec4079be-b302-497b-9e34-abe0e7208507\") "
Oct 30 14:10:40.874374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:40.874338    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-openshift-ca" pod "ec4079be-b302-497b-9e34-abe0e7208507" (UID: "ec4079be-b302-497b-9e34-abe0e7208507"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:40.874838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:40.874819    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.876280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:40.876254    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4079be-b302-497b-9e34-abe0e7208507-kube-api-access-5mtv8" pod "ec4079be-b302-497b-9e34-abe0e7208507" (UID: "ec4079be-b302-497b-9e34-abe0e7208507"). InnerVolumeSpecName "kube-api-access-5mtv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:40.876956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:40.876934    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-xks-ca" pod "ec4079be-b302-497b-9e34-abe0e7208507" (UID: "ec4079be-b302-497b-9e34-abe0e7208507"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:40.976177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:40.976109    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mtv8\" (UniqueName: \"kubernetes.io/projected/ec4079be-b302-497b-9e34-abe0e7208507-kube-api-access-5mtv8\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.976177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:40.976141    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4079be-b302-497b-9e34-abe0e7208507-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:41.367151 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:10:41.367118    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:10:41.659668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.659576    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr" event={"ID":"ec4079be-b302-497b-9e34-abe0e7208507","Type":"ContainerDied","Data":"0f5e5a93ab7baa4a9fd3df65de7273a99cd1e7023d961d046f7d2c4983b191c6"}
Oct 30 14:10:41.659668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.659612    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"
Oct 30 14:10:41.659668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.659641    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f5e5a93ab7baa4a9fd3df65de7273a99cd1e7023d961d046f7d2c4983b191c6"
Oct 30 14:10:41.928226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.928143    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"]
Oct 30 14:10:41.928820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.928806    4530 cpu_manager.go:414] "containerMap: removing container" podUID="ec4079be-b302-497b-9e34-abe0e7208507" containerName="revoke-keys"
Oct 30 14:10:41.928876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.928823    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4079be-b302-497b-9e34-abe0e7208507" containerName="revoke-keys"
Oct 30 14:10:41.928974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.928964    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4079be-b302-497b-9e34-abe0e7208507" containerName="revoke-keys"
Oct 30 14:10:41.932858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.932841    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:41.950978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.950948    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"]
Oct 30 14:10:41.951160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.951137    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:10:41.951284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.951245    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:10:41.999389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.999360    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4f933cc-hck22\" (UID: \"7b7328ad-201c-41b8-b9c8-a32423392f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:41.999589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.999531    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4f933cc-hck22\" (UID: \"7b7328ad-201c-41b8-b9c8-a32423392f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:41.999720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:41.999705    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9z9k\" (UniqueName: \"kubernetes.io/projected/7b7328ad-201c-41b8-b9c8-a32423392f40-kube-api-access-s9z9k\") pod \"maas-api-revoke-sub-d4f933cc-hck22\" (UID: \"7b7328ad-201c-41b8-b9c8-a32423392f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:42.101609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:42.101577    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4f933cc-hck22\" (UID: \"7b7328ad-201c-41b8-b9c8-a32423392f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:42.101760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:42.101726    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4f933cc-hck22\" (UID: \"7b7328ad-201c-41b8-b9c8-a32423392f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:42.101861 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:42.101844    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9z9k\" (UniqueName: \"kubernetes.io/projected/7b7328ad-201c-41b8-b9c8-a32423392f40-kube-api-access-s9z9k\") pod \"maas-api-revoke-sub-d4f933cc-hck22\" (UID: \"7b7328ad-201c-41b8-b9c8-a32423392f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:42.102318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:42.102291    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4f933cc-hck22\" (UID: \"7b7328ad-201c-41b8-b9c8-a32423392f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:42.104076 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:42.104055    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4f933cc-hck22\" (UID: \"7b7328ad-201c-41b8-b9c8-a32423392f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:42.128551 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:42.128519    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9z9k\" (UniqueName: \"kubernetes.io/projected/7b7328ad-201c-41b8-b9c8-a32423392f40-kube-api-access-s9z9k\") pod \"maas-api-revoke-sub-d4f933cc-hck22\" (UID: \"7b7328ad-201c-41b8-b9c8-a32423392f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:42.248044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:42.247961    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:42.708007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:42.707896    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"]
Oct 30 14:10:42.719731 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:10:42.718228    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7328ad_201c_41b8_b9c8_a32423392f40.slice/crio-ba154078a45f785ac9ee04e22475e95e4526c798a75a71350e02a259b3467aa0 WatchSource:0}: Error finding container ba154078a45f785ac9ee04e22475e95e4526c798a75a71350e02a259b3467aa0: Status 404 returned error can't find the container with id ba154078a45f785ac9ee04e22475e95e4526c798a75a71350e02a259b3467aa0
Oct 30 14:10:43.704598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:43.704566    4530 generic.go:356] "Generic (PLEG): container finished" podID="7b7328ad-201c-41b8-b9c8-a32423392f40" containerID="56f222e6a5a9562bb5d77380bf80ca5fb2f9946637d61717ff76a1bf67910006" exitCode=0
Oct 30 14:10:43.704935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:43.704613    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22" event={"ID":"7b7328ad-201c-41b8-b9c8-a32423392f40","Type":"ContainerDied","Data":"56f222e6a5a9562bb5d77380bf80ca5fb2f9946637d61717ff76a1bf67910006"}
Oct 30 14:10:43.704935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:43.704674    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22" event={"ID":"7b7328ad-201c-41b8-b9c8-a32423392f40","Type":"ContainerStarted","Data":"ba154078a45f785ac9ee04e22475e95e4526c798a75a71350e02a259b3467aa0"}
Oct 30 14:10:44.868499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:44.868474    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:44.980250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:44.980172    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-openshift-ca\") pod \"7b7328ad-201c-41b8-b9c8-a32423392f40\" (UID: \"7b7328ad-201c-41b8-b9c8-a32423392f40\") "
Oct 30 14:10:44.980392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:44.980268    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-xks-ca\") pod \"7b7328ad-201c-41b8-b9c8-a32423392f40\" (UID: \"7b7328ad-201c-41b8-b9c8-a32423392f40\") "
Oct 30 14:10:44.980452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:44.980438    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7328ad-201c-41b8-b9c8-a32423392f40-kube-api-access-s9z9k\" (UniqueName: \"kubernetes.io/projected/7b7328ad-201c-41b8-b9c8-a32423392f40-kube-api-access-s9z9k\") pod \"7b7328ad-201c-41b8-b9c8-a32423392f40\" (UID: \"7b7328ad-201c-41b8-b9c8-a32423392f40\") "
Oct 30 14:10:44.980533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:44.980509    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-openshift-ca" pod "7b7328ad-201c-41b8-b9c8-a32423392f40" (UID: "7b7328ad-201c-41b8-b9c8-a32423392f40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:44.980872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:44.980849    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:44.982410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:44.982381    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-xks-ca" pod "7b7328ad-201c-41b8-b9c8-a32423392f40" (UID: "7b7328ad-201c-41b8-b9c8-a32423392f40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:44.982495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:44.982439    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7328ad-201c-41b8-b9c8-a32423392f40-kube-api-access-s9z9k" pod "7b7328ad-201c-41b8-b9c8-a32423392f40" (UID: "7b7328ad-201c-41b8-b9c8-a32423392f40"). InnerVolumeSpecName "kube-api-access-s9z9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:45.082728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:45.082698    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7328ad-201c-41b8-b9c8-a32423392f40-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:45.082728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:45.082730    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9z9k\" (UniqueName: \"kubernetes.io/projected/7b7328ad-201c-41b8-b9c8-a32423392f40-kube-api-access-s9z9k\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:45.721863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:45.721832    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"
Oct 30 14:10:45.722108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:45.721870    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22" event={"ID":"7b7328ad-201c-41b8-b9c8-a32423392f40","Type":"ContainerDied","Data":"ba154078a45f785ac9ee04e22475e95e4526c798a75a71350e02a259b3467aa0"}
Oct 30 14:10:45.722108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:45.721921    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba154078a45f785ac9ee04e22475e95e4526c798a75a71350e02a259b3467aa0"
Oct 30 14:10:47.126506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:47.126476    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:47.138254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:47.138226    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:10:48.988706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:48.988623    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"]
Oct 30 14:10:48.990278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:48.990248    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7b7328ad-201c-41b8-b9c8-a32423392f40" containerName="revoke-keys"
Oct 30 14:10:48.990451 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:48.990436    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7328ad-201c-41b8-b9c8-a32423392f40" containerName="revoke-keys"
Oct 30 14:10:48.990811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:48.990794    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7328ad-201c-41b8-b9c8-a32423392f40" containerName="revoke-keys"
Oct 30 14:10:49.018796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.018756    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"]
Oct 30 14:10:49.019740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.019648    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:49.063121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.063089    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:10:49.063303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.063148    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:10:49.157818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.157777    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a63741b-c431-4056-94d2-a996861a0388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6824b7ba-ngckn\" (UID: \"4a63741b-c431-4056-94d2-a996861a0388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:49.157992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.157894    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a63741b-c431-4056-94d2-a996861a0388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6824b7ba-ngckn\" (UID: \"4a63741b-c431-4056-94d2-a996861a0388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:49.158064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.158038    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cnxx\" (UniqueName: \"kubernetes.io/projected/4a63741b-c431-4056-94d2-a996861a0388-kube-api-access-2cnxx\") pod \"maas-api-revoke-sub-6824b7ba-ngckn\" (UID: \"4a63741b-c431-4056-94d2-a996861a0388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:49.259827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.259752    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cnxx\" (UniqueName: \"kubernetes.io/projected/4a63741b-c431-4056-94d2-a996861a0388-kube-api-access-2cnxx\") pod \"maas-api-revoke-sub-6824b7ba-ngckn\" (UID: \"4a63741b-c431-4056-94d2-a996861a0388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:49.260015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.259880    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a63741b-c431-4056-94d2-a996861a0388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6824b7ba-ngckn\" (UID: \"4a63741b-c431-4056-94d2-a996861a0388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:49.260015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.259929    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a63741b-c431-4056-94d2-a996861a0388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6824b7ba-ngckn\" (UID: \"4a63741b-c431-4056-94d2-a996861a0388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:49.260619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.260595    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a63741b-c431-4056-94d2-a996861a0388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6824b7ba-ngckn\" (UID: \"4a63741b-c431-4056-94d2-a996861a0388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:49.262257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.262234    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a63741b-c431-4056-94d2-a996861a0388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6824b7ba-ngckn\" (UID: \"4a63741b-c431-4056-94d2-a996861a0388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:49.286351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.286327    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cnxx\" (UniqueName: \"kubernetes.io/projected/4a63741b-c431-4056-94d2-a996861a0388-kube-api-access-2cnxx\") pod \"maas-api-revoke-sub-6824b7ba-ngckn\" (UID: \"4a63741b-c431-4056-94d2-a996861a0388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:49.371857 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.371833    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:49.710686 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:10:49.710651    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a63741b_c431_4056_94d2_a996861a0388.slice/crio-1735b0c4969e1d7c07501060c49727a33c455e26da9ccc8e9225197333393ff6 WatchSource:0}: Error finding container 1735b0c4969e1d7c07501060c49727a33c455e26da9ccc8e9225197333393ff6: Status 404 returned error can't find the container with id 1735b0c4969e1d7c07501060c49727a33c455e26da9ccc8e9225197333393ff6
Oct 30 14:10:49.714011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.713986    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"]
Oct 30 14:10:49.755709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:49.755681    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn" event={"ID":"4a63741b-c431-4056-94d2-a996861a0388","Type":"ContainerStarted","Data":"1735b0c4969e1d7c07501060c49727a33c455e26da9ccc8e9225197333393ff6"}
Oct 30 14:10:50.829430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:50.829387    4530 generic.go:356] "Generic (PLEG): container finished" podID="4a63741b-c431-4056-94d2-a996861a0388" containerID="62775a57688a2fd1a6c02e228254ab965072b215ce46b03ecd0f7ea9fa4fadc3" exitCode=0
Oct 30 14:10:50.830096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:50.830066    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn" event={"ID":"4a63741b-c431-4056-94d2-a996861a0388","Type":"ContainerDied","Data":"62775a57688a2fd1a6c02e228254ab965072b215ce46b03ecd0f7ea9fa4fadc3"}
Oct 30 14:10:52.006401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.006377    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:52.112087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.112008    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a63741b-c431-4056-94d2-a996861a0388-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a63741b-c431-4056-94d2-a996861a0388-maas-api-xks-ca\") pod \"4a63741b-c431-4056-94d2-a996861a0388\" (UID: \"4a63741b-c431-4056-94d2-a996861a0388\") "
Oct 30 14:10:52.112087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.112069    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a63741b-c431-4056-94d2-a996861a0388-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a63741b-c431-4056-94d2-a996861a0388-maas-api-openshift-ca\") pod \"4a63741b-c431-4056-94d2-a996861a0388\" (UID: \"4a63741b-c431-4056-94d2-a996861a0388\") "
Oct 30 14:10:52.112271 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.112153    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a63741b-c431-4056-94d2-a996861a0388-kube-api-access-2cnxx\" (UniqueName: \"kubernetes.io/projected/4a63741b-c431-4056-94d2-a996861a0388-kube-api-access-2cnxx\") pod \"4a63741b-c431-4056-94d2-a996861a0388\" (UID: \"4a63741b-c431-4056-94d2-a996861a0388\") "
Oct 30 14:10:52.112485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.112458    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a63741b-c431-4056-94d2-a996861a0388-maas-api-openshift-ca" pod "4a63741b-c431-4056-94d2-a996861a0388" (UID: "4a63741b-c431-4056-94d2-a996861a0388"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:52.112644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.112627    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a63741b-c431-4056-94d2-a996861a0388-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:52.114271 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.114237    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a63741b-c431-4056-94d2-a996861a0388-kube-api-access-2cnxx" pod "4a63741b-c431-4056-94d2-a996861a0388" (UID: "4a63741b-c431-4056-94d2-a996861a0388"). InnerVolumeSpecName "kube-api-access-2cnxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:52.114400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.114286    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a63741b-c431-4056-94d2-a996861a0388-maas-api-xks-ca" pod "4a63741b-c431-4056-94d2-a996861a0388" (UID: "4a63741b-c431-4056-94d2-a996861a0388"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:52.214174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.214142    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a63741b-c431-4056-94d2-a996861a0388-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:52.214174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.214175    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cnxx\" (UniqueName: \"kubernetes.io/projected/4a63741b-c431-4056-94d2-a996861a0388-kube-api-access-2cnxx\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:52.414440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.414370    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"]
Oct 30 14:10:52.415141 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.415124    4530 cpu_manager.go:414] "containerMap: removing container" podUID="4a63741b-c431-4056-94d2-a996861a0388" containerName="revoke-keys"
Oct 30 14:10:52.415196 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.415143    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a63741b-c431-4056-94d2-a996861a0388" containerName="revoke-keys"
Oct 30 14:10:52.415269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.415260    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a63741b-c431-4056-94d2-a996861a0388" containerName="revoke-keys"
Oct 30 14:10:52.419240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.419213    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:52.435649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.435620    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"]
Oct 30 14:10:52.518053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.518013    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34a976ca-z99px\" (UID: \"c73d6c78-38f5-4172-a216-f511ac326c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:52.518279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.518262    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwrd6\" (UniqueName: \"kubernetes.io/projected/c73d6c78-38f5-4172-a216-f511ac326c95-kube-api-access-cwrd6\") pod \"maas-api-revoke-sub-34a976ca-z99px\" (UID: \"c73d6c78-38f5-4172-a216-f511ac326c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:52.518343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.518329    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34a976ca-z99px\" (UID: \"c73d6c78-38f5-4172-a216-f511ac326c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:52.622590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.620233    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrd6\" (UniqueName: \"kubernetes.io/projected/c73d6c78-38f5-4172-a216-f511ac326c95-kube-api-access-cwrd6\") pod \"maas-api-revoke-sub-34a976ca-z99px\" (UID: \"c73d6c78-38f5-4172-a216-f511ac326c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:52.622590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.620325    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34a976ca-z99px\" (UID: \"c73d6c78-38f5-4172-a216-f511ac326c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:52.622590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.620488    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34a976ca-z99px\" (UID: \"c73d6c78-38f5-4172-a216-f511ac326c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:52.622590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.621509    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34a976ca-z99px\" (UID: \"c73d6c78-38f5-4172-a216-f511ac326c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:52.641636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.639997    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34a976ca-z99px\" (UID: \"c73d6c78-38f5-4172-a216-f511ac326c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:52.650716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.650650    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrd6\" (UniqueName: \"kubernetes.io/projected/c73d6c78-38f5-4172-a216-f511ac326c95-kube-api-access-cwrd6\") pod \"maas-api-revoke-sub-34a976ca-z99px\" (UID: \"c73d6c78-38f5-4172-a216-f511ac326c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:52.734942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.734866    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:52.853096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.853058    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn" event={"ID":"4a63741b-c431-4056-94d2-a996861a0388","Type":"ContainerDied","Data":"1735b0c4969e1d7c07501060c49727a33c455e26da9ccc8e9225197333393ff6"}
Oct 30 14:10:52.853221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.853124    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1735b0c4969e1d7c07501060c49727a33c455e26da9ccc8e9225197333393ff6"
Oct 30 14:10:52.853221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:52.853170    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"
Oct 30 14:10:53.071920 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:10:53.071886    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73d6c78_38f5_4172_a216_f511ac326c95.slice/crio-4e1ce12ae6bdb757b6a33c4afd0324384b5eb9a7b752e21c4f732b0155b28cc3 WatchSource:0}: Error finding container 4e1ce12ae6bdb757b6a33c4afd0324384b5eb9a7b752e21c4f732b0155b28cc3: Status 404 returned error can't find the container with id 4e1ce12ae6bdb757b6a33c4afd0324384b5eb9a7b752e21c4f732b0155b28cc3
Oct 30 14:10:53.075534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:53.075515    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"]
Oct 30 14:10:53.863092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:53.863060    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px" event={"ID":"c73d6c78-38f5-4172-a216-f511ac326c95","Type":"ContainerStarted","Data":"e7432b6dc762377ff4567d967bbd2680aa8e9e6759f7a8c57f3590f809effcae"}
Oct 30 14:10:53.863290 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:53.863114    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px" event={"ID":"c73d6c78-38f5-4172-a216-f511ac326c95","Type":"ContainerStarted","Data":"4e1ce12ae6bdb757b6a33c4afd0324384b5eb9a7b752e21c4f732b0155b28cc3"}
Oct 30 14:10:53.911373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:53.911322    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px" podStartSLOduration=1.911308843 podStartE2EDuration="1.911308843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:53.9091034 +0000 UTC m=+1318.941002342" watchObservedRunningTime="2026-10-30 14:10:53.911308843 +0000 UTC m=+1318.943207782"
Oct 30 14:10:54.873117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:54.873080    4530 generic.go:356] "Generic (PLEG): container finished" podID="c73d6c78-38f5-4172-a216-f511ac326c95" containerID="e7432b6dc762377ff4567d967bbd2680aa8e9e6759f7a8c57f3590f809effcae" exitCode=0
Oct 30 14:10:54.873506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:54.873163    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px" event={"ID":"c73d6c78-38f5-4172-a216-f511ac326c95","Type":"ContainerDied","Data":"e7432b6dc762377ff4567d967bbd2680aa8e9e6759f7a8c57f3590f809effcae"}
Oct 30 14:10:55.703838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.703807    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"]
Oct 30 14:10:55.708293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.708269    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.731997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.731968    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"]
Oct 30 14:10:55.768157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.768118    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.768331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.768246    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.768331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.768303    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.768456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.768412    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.768456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.768447    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.768594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.768533    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.768758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.768731    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.768828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.768775    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.768901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.768886    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c687\" (UniqueName: \"kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-kube-api-access-2c687\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870005    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870055    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870093    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870110    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870134    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870233    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870348    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870476 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870416    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c687\" (UniqueName: \"kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-kube-api-access-2c687\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870510    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870569    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870629    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870722    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.870957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870935    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.871027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.870992    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.872599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.872581    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.872795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.872777    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.896838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.896816    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c687\" (UniqueName: \"kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-kube-api-access-2c687\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:55.897155 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:55.896914    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:56.028440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.028411    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:56.034894 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.034875    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:56.230502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.210513    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-openshift-ca\") pod \"c73d6c78-38f5-4172-a216-f511ac326c95\" (UID: \"c73d6c78-38f5-4172-a216-f511ac326c95\") "
Oct 30 14:10:56.230502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.210735    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-xks-ca\") pod \"c73d6c78-38f5-4172-a216-f511ac326c95\" (UID: \"c73d6c78-38f5-4172-a216-f511ac326c95\") "
Oct 30 14:10:56.230502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.210914    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73d6c78-38f5-4172-a216-f511ac326c95-kube-api-access-cwrd6\" (UniqueName: \"kubernetes.io/projected/c73d6c78-38f5-4172-a216-f511ac326c95-kube-api-access-cwrd6\") pod \"c73d6c78-38f5-4172-a216-f511ac326c95\" (UID: \"c73d6c78-38f5-4172-a216-f511ac326c95\") "
Oct 30 14:10:56.231419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.231023    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-openshift-ca" pod "c73d6c78-38f5-4172-a216-f511ac326c95" (UID: "c73d6c78-38f5-4172-a216-f511ac326c95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:56.273941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.273811    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73d6c78-38f5-4172-a216-f511ac326c95-kube-api-access-cwrd6" pod "c73d6c78-38f5-4172-a216-f511ac326c95" (UID: "c73d6c78-38f5-4172-a216-f511ac326c95"). InnerVolumeSpecName "kube-api-access-cwrd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:56.294757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.294695    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-xks-ca" pod "c73d6c78-38f5-4172-a216-f511ac326c95" (UID: "c73d6c78-38f5-4172-a216-f511ac326c95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:56.330529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.329491    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:56.330529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.329522    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwrd6\" (UniqueName: \"kubernetes.io/projected/c73d6c78-38f5-4172-a216-f511ac326c95-kube-api-access-cwrd6\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:56.330529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.329579    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73d6c78-38f5-4172-a216-f511ac326c95-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:10:56.372746 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:10:56.372714    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb0ff55_8edc_43e6_b24e_6325a97efd1c.slice/crio-fd3722400e74a621471e03ac7b9c4227e27684e7cf3dd68af8b8badb90061c15 WatchSource:0}: Error finding container fd3722400e74a621471e03ac7b9c4227e27684e7cf3dd68af8b8badb90061c15: Status 404 returned error can't find the container with id fd3722400e74a621471e03ac7b9c4227e27684e7cf3dd68af8b8badb90061c15
Oct 30 14:10:56.375086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.375050    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:10:56.375178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.375121    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:10:56.375178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.375150    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:10:56.375273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.375253    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"]
Oct 30 14:10:56.797888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.797842    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"]
Oct 30 14:10:56.799004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.798975    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c73d6c78-38f5-4172-a216-f511ac326c95" containerName="revoke-keys"
Oct 30 14:10:56.799004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.799000    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73d6c78-38f5-4172-a216-f511ac326c95" containerName="revoke-keys"
Oct 30 14:10:56.799241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.799226    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73d6c78-38f5-4172-a216-f511ac326c95" containerName="revoke-keys"
Oct 30 14:10:56.805278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.805255    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.819579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.819554    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"]
Oct 30 14:10:56.826046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.826026    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"distinct-2-w0-2df0ab-kserve-self-signed-certs\""
Oct 30 14:10:56.837802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.837778    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tls-certs\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.837929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.837817    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tmp-dir\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.837996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.837946    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-home\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.838113 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.838097    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vttht\" (UniqueName: \"kubernetes.io/projected/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-kube-api-access-vttht\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.838181 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.838168    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-dshm\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.838260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.838248    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-model-cache\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.900684 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.900645    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" event={"ID":"cfb0ff55-8edc-43e6-b24e-6325a97efd1c","Type":"ContainerStarted","Data":"8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50"}
Oct 30 14:10:56.901075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.900746    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" event={"ID":"cfb0ff55-8edc-43e6-b24e-6325a97efd1c","Type":"ContainerStarted","Data":"fd3722400e74a621471e03ac7b9c4227e27684e7cf3dd68af8b8badb90061c15"}
Oct 30 14:10:56.902229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.902209    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"
Oct 30 14:10:56.902302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.902235    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px" event={"ID":"c73d6c78-38f5-4172-a216-f511ac326c95","Type":"ContainerDied","Data":"4e1ce12ae6bdb757b6a33c4afd0324384b5eb9a7b752e21c4f732b0155b28cc3"}
Oct 30 14:10:56.902302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.902270    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e1ce12ae6bdb757b6a33c4afd0324384b5eb9a7b752e21c4f732b0155b28cc3"
Oct 30 14:10:56.939330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.939303    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-model-cache\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.939434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.939392    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tls-certs\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.939434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.939415    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tmp-dir\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.939575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.939455    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-home\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.939684 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.939662    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-model-cache\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.939773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.939754    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tmp-dir\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.939843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.939783    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vttht\" (UniqueName: \"kubernetes.io/projected/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-kube-api-access-vttht\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.939922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.939904    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-dshm\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.939991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.939959    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-home\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.941942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.941920    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tls-certs\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.942029 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.941927    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-dshm\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:56.953706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:56.953658    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podStartSLOduration=1.9536424669999999 podStartE2EDuration="1.953642467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:56.949955268 +0000 UTC m=+1321.981854209" watchObservedRunningTime="2026-10-30 14:10:56.953642467 +0000 UTC m=+1321.985541407"
Oct 30 14:10:57.012576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:57.010279    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vttht\" (UniqueName: \"kubernetes.io/projected/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-kube-api-access-vttht\") pod \"distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:57.036562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:57.036185    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:10:57.036562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:57.036305    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:10:57.036562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:57.036348    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:10:57.130211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:57.130128    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:57.557103 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:10:57.557072    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2893d528_7fcc_456f_bf7f_61f7d7f89b2a.slice/crio-f501b8518befc2f1f3449a0b72ecaa6c2e15b2bb30a6809dd1eef92d3d24cc6e WatchSource:0}: Error finding container f501b8518befc2f1f3449a0b72ecaa6c2e15b2bb30a6809dd1eef92d3d24cc6e: Status 404 returned error can't find the container with id f501b8518befc2f1f3449a0b72ecaa6c2e15b2bb30a6809dd1eef92d3d24cc6e
Oct 30 14:10:57.560442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:57.560415    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"]
Oct 30 14:10:57.915525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:57.915425    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" event={"ID":"2893d528-7fcc-456f-bf7f-61f7d7f89b2a","Type":"ContainerStarted","Data":"2822643bb78b9d408ad6c4c5246aed382fe8c9ae6e29bd56a08cb04323f1fd8a"}
Oct 30 14:10:57.915525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:57.915478    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" event={"ID":"2893d528-7fcc-456f-bf7f-61f7d7f89b2a","Type":"ContainerStarted","Data":"f501b8518befc2f1f3449a0b72ecaa6c2e15b2bb30a6809dd1eef92d3d24cc6e"}
Oct 30 14:10:57.962911 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:57.962868    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podStartSLOduration=1.962850686 podStartE2EDuration="1.962850686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:57.961476009 +0000 UTC m=+1322.993374948" watchObservedRunningTime="2026-10-30 14:10:57.962850686 +0000 UTC m=+1322.994749624"
Oct 30 14:10:58.029630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:58.029595    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:10:58.029805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:58.029644    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:10:58.148564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:58.145647    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:10:59.029740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:59.029707    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:10:59.030099 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:10:59.029765    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:11:00.029205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:00.029173    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:11:00.029374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:00.029219    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:11:01.090621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:01.082832    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:11:01.090621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:01.082913    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:11:02.027877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:02.026647    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:02.029135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:02.029095    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:11:02.029331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:02.029308    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:11:02.036204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:02.036163    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:02.130071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:02.130002    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:02.409710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:02.409667    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:02.469679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:02.469632    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:03.058636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:03.053140    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:11:03.058636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:03.053200    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:11:03.060699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:03.060659    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 14:11:03.079322 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:03.061253    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:11:03.212816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:03.212766    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:03.444529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:03.420706    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:03.946595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:03.917954    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:03.946595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:03.918019    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:03.946595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:03.925383    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:11:03.946595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:03.925480    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:11:04.021582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.020211    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:04.021582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.020270    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:04.085585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.050766    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:11:04.085585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.050830    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:11:04.085585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.083908    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:04.146596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.107048    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:04.146596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.107117    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" podUID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:04.260937 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.260834    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:04.334582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.334053    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:04.334582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.334122    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:04.335367 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.335317    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:04.335603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.335579    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:04.456781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.456732    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:04.779570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.777082    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:04.779570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:04.777148    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:05.072651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:05.049802    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:11:05.072651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:05.049860    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:11:05.087483 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:05.087438    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:05.201727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:05.201690    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:05.202004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:05.201972    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:05.367642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:05.349595    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:11:05.505575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:05.501353    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Oct 30 14:11:06.041577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:06.041062    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:11:06.041577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:06.041160    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:11:06.093567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:06.092738    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:06.375753 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:06.375707    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:06.535574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:06.502315    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:07.036228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:07.036184    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:11:07.036420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:07.036259    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:11:07.096429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:07.096382    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:07.269567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:07.265566    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:07.269567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:07.265629    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:07.383627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:07.378631    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:11:07.517498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:07.500083    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:07.517498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:07.500144    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:07.517498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:07.512671    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:07.535584 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:07.535477    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:11:07.931786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:07.931609    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:11:07.954118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:07.954088    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:11:08.022789 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:08.022761    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:11:08.053425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:08.053393    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:11:08.053707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:08.053674    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:11:08.056930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:08.056907    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:11:08.366493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:08.365966    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:11:09.029301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:09.029263    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 14:11:09.029799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:09.029321    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 14:11:10.274555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:10.274509    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:11:10.275054 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:10.274991    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:11:10.275872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:10.275851    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:11:11.033480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.033449    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"]
Oct 30 14:11:11.039208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.039191    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:11.057582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.057530    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"]
Oct 30 14:11:11.065502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.065481    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:11:11.066009 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.065987    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:11:11.105531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.105506    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56011259-8cjsm\" (UID: \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:11.105691 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.105587    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd9fz\" (UniqueName: \"kubernetes.io/projected/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-kube-api-access-kd9fz\") pod \"maas-api-revoke-sub-56011259-8cjsm\" (UID: \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:11.105742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.105709    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56011259-8cjsm\" (UID: \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:11.207623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.207590    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56011259-8cjsm\" (UID: \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:11.207775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.207636    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9fz\" (UniqueName: \"kubernetes.io/projected/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-kube-api-access-kd9fz\") pod \"maas-api-revoke-sub-56011259-8cjsm\" (UID: \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:11.207859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.207771    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56011259-8cjsm\" (UID: \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:11.208319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.208295    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56011259-8cjsm\" (UID: \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:11.209848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.209824    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56011259-8cjsm\" (UID: \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:11.234243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.234219    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd9fz\" (UniqueName: \"kubernetes.io/projected/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-kube-api-access-kd9fz\") pod \"maas-api-revoke-sub-56011259-8cjsm\" (UID: \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:11.365573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.365521    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:11.367431 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:11:11.367399    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:11:11.668890 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:11:11.668856    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1fbf64_5e02_43ef_a805_7b7d0aa19ce7.slice/crio-be64e6a73121697d59bf255c500c88be78d5c77dc67724496db24a3c31bf15f6 WatchSource:0}: Error finding container be64e6a73121697d59bf255c500c88be78d5c77dc67724496db24a3c31bf15f6: Status 404 returned error can't find the container with id be64e6a73121697d59bf255c500c88be78d5c77dc67724496db24a3c31bf15f6
Oct 30 14:11:11.672744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:11.672715    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"]
Oct 30 14:11:12.521900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:12.521858    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm" event={"ID":"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7","Type":"ContainerStarted","Data":"7045891e5108d81bf5bc467078b61bd0e50c38d8cebd88b274a9b7aef8115345"}
Oct 30 14:11:12.523465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:12.521927    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm" event={"ID":"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7","Type":"ContainerStarted","Data":"be64e6a73121697d59bf255c500c88be78d5c77dc67724496db24a3c31bf15f6"}
Oct 30 14:11:12.570529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:12.570484    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm" podStartSLOduration=1.5704710560000001 podStartE2EDuration="1.570471056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:12.568207214 +0000 UTC m=+1337.600106153" watchObservedRunningTime="2026-10-30 14:11:12.570471056 +0000 UTC m=+1337.602369994"
Oct 30 14:11:13.531050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:13.531013    4530 generic.go:356] "Generic (PLEG): container finished" podID="cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7" containerID="7045891e5108d81bf5bc467078b61bd0e50c38d8cebd88b274a9b7aef8115345" exitCode=0
Oct 30 14:11:13.531410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:13.531100    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm" event={"ID":"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7","Type":"ContainerDied","Data":"7045891e5108d81bf5bc467078b61bd0e50c38d8cebd88b274a9b7aef8115345"}
Oct 30 14:11:14.697072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:14.697046    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:14.768756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:14.768722    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-openshift-ca\") pod \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\" (UID: \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\") "
Oct 30 14:11:14.768944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:14.768830    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-xks-ca\") pod \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\" (UID: \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\") "
Oct 30 14:11:14.769016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:14.768971    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-kube-api-access-kd9fz\" (UniqueName: \"kubernetes.io/projected/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-kube-api-access-kd9fz\") pod \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\" (UID: \"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7\") "
Oct 30 14:11:14.769143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:14.769118    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-openshift-ca" pod "cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7" (UID: "cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:14.769870 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:14.769840    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:14.771180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:14.771154    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-kube-api-access-kd9fz" pod "cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7" (UID: "cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7"). InnerVolumeSpecName "kube-api-access-kd9fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:14.771352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:14.771332    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-xks-ca" pod "cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7" (UID: "cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:14.871354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:14.871330    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:14.871354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:14.871355    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd9fz\" (UniqueName: \"kubernetes.io/projected/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7-kube-api-access-kd9fz\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:15.548343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:15.548310    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"
Oct 30 14:11:15.548343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:15.548320    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm" event={"ID":"cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7","Type":"ContainerDied","Data":"be64e6a73121697d59bf255c500c88be78d5c77dc67724496db24a3c31bf15f6"}
Oct 30 14:11:15.548569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:15.548383    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be64e6a73121697d59bf255c500c88be78d5c77dc67724496db24a3c31bf15f6"
Oct 30 14:11:15.971116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:15.971030    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"]
Oct 30 14:11:15.971973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:15.971951    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7" containerName="revoke-keys"
Oct 30 14:11:15.971973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:15.971973    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7" containerName="revoke-keys"
Oct 30 14:11:15.972147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:15.972134    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7" containerName="revoke-keys"
Oct 30 14:11:15.975967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:15.975945    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:15.997115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:15.997090    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"]
Oct 30 14:11:16.004092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.004072    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:11:16.004490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.004472    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:11:16.088888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.087849    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae8dcb50-58x5w\" (UID: \"b9ca9125-32d2-4407-b359-7f3d56602f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:16.089385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.089354    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwss2\" (UniqueName: \"kubernetes.io/projected/b9ca9125-32d2-4407-b359-7f3d56602f59-kube-api-access-qwss2\") pod \"maas-api-revoke-sub-ae8dcb50-58x5w\" (UID: \"b9ca9125-32d2-4407-b359-7f3d56602f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:16.089817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.089758    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae8dcb50-58x5w\" (UID: \"b9ca9125-32d2-4407-b359-7f3d56602f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:16.192041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.191914    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae8dcb50-58x5w\" (UID: \"b9ca9125-32d2-4407-b359-7f3d56602f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:16.192505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.192476    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae8dcb50-58x5w\" (UID: \"b9ca9125-32d2-4407-b359-7f3d56602f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:16.192815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.192781    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwss2\" (UniqueName: \"kubernetes.io/projected/b9ca9125-32d2-4407-b359-7f3d56602f59-kube-api-access-qwss2\") pod \"maas-api-revoke-sub-ae8dcb50-58x5w\" (UID: \"b9ca9125-32d2-4407-b359-7f3d56602f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:16.195343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.195321    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae8dcb50-58x5w\" (UID: \"b9ca9125-32d2-4407-b359-7f3d56602f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:16.198313 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.198267    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae8dcb50-58x5w\" (UID: \"b9ca9125-32d2-4407-b359-7f3d56602f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:16.231989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.231894    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwss2\" (UniqueName: \"kubernetes.io/projected/b9ca9125-32d2-4407-b359-7f3d56602f59-kube-api-access-qwss2\") pod \"maas-api-revoke-sub-ae8dcb50-58x5w\" (UID: \"b9ca9125-32d2-4407-b359-7f3d56602f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:16.298853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.298826    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:16.612330 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:11:16.612304    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ca9125_32d2_4407_b359_7f3d56602f59.slice/crio-c4a7daf4f34dd0d5556382b3992b9760f324bf381f565f56e0c3f36be5dd8227 WatchSource:0}: Error finding container c4a7daf4f34dd0d5556382b3992b9760f324bf381f565f56e0c3f36be5dd8227: Status 404 returned error can't find the container with id c4a7daf4f34dd0d5556382b3992b9760f324bf381f565f56e0c3f36be5dd8227
Oct 30 14:11:16.615824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:16.615802    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"]
Oct 30 14:11:17.565775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:17.565744    4530 generic.go:356] "Generic (PLEG): container finished" podID="b9ca9125-32d2-4407-b359-7f3d56602f59" containerID="6b5f084ff1d1b3217b29182e201049e8ab710e8b2d3270f33b734e094a75e870" exitCode=0
Oct 30 14:11:17.566250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:17.565828    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w" event={"ID":"b9ca9125-32d2-4407-b359-7f3d56602f59","Type":"ContainerDied","Data":"6b5f084ff1d1b3217b29182e201049e8ab710e8b2d3270f33b734e094a75e870"}
Oct 30 14:11:17.566250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:17.565877    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w" event={"ID":"b9ca9125-32d2-4407-b359-7f3d56602f59","Type":"ContainerStarted","Data":"c4a7daf4f34dd0d5556382b3992b9760f324bf381f565f56e0c3f36be5dd8227"}
Oct 30 14:11:18.741085 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.741055    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:18.847721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.847621    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-xks-ca\") pod \"b9ca9125-32d2-4407-b359-7f3d56602f59\" (UID: \"b9ca9125-32d2-4407-b359-7f3d56602f59\") "
Oct 30 14:11:18.847900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.847737    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-openshift-ca\") pod \"b9ca9125-32d2-4407-b359-7f3d56602f59\" (UID: \"b9ca9125-32d2-4407-b359-7f3d56602f59\") "
Oct 30 14:11:18.847973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.847933    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ca9125-32d2-4407-b359-7f3d56602f59-kube-api-access-qwss2\" (UniqueName: \"kubernetes.io/projected/b9ca9125-32d2-4407-b359-7f3d56602f59-kube-api-access-qwss2\") pod \"b9ca9125-32d2-4407-b359-7f3d56602f59\" (UID: \"b9ca9125-32d2-4407-b359-7f3d56602f59\") "
Oct 30 14:11:18.848529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.848174    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-openshift-ca" pod "b9ca9125-32d2-4407-b359-7f3d56602f59" (UID: "b9ca9125-32d2-4407-b359-7f3d56602f59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:18.848721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.848706    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:18.850108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.850078    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-xks-ca" pod "b9ca9125-32d2-4407-b359-7f3d56602f59" (UID: "b9ca9125-32d2-4407-b359-7f3d56602f59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:18.850802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.850775    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ca9125-32d2-4407-b359-7f3d56602f59-kube-api-access-qwss2" pod "b9ca9125-32d2-4407-b359-7f3d56602f59" (UID: "b9ca9125-32d2-4407-b359-7f3d56602f59"). InnerVolumeSpecName "kube-api-access-qwss2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:18.936498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.936465    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"]
Oct 30 14:11:18.937150 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.937135    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b9ca9125-32d2-4407-b359-7f3d56602f59" containerName="revoke-keys"
Oct 30 14:11:18.937198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.937150    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ca9125-32d2-4407-b359-7f3d56602f59" containerName="revoke-keys"
Oct 30 14:11:18.937289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.937280    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ca9125-32d2-4407-b359-7f3d56602f59" containerName="revoke-keys"
Oct 30 14:11:18.941135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.941119    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:18.950199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.950172    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ca9125-32d2-4407-b359-7f3d56602f59-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:18.950337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.950201    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwss2\" (UniqueName: \"kubernetes.io/projected/b9ca9125-32d2-4407-b359-7f3d56602f59-kube-api-access-qwss2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:18.958976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:18.958950    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"]
Oct 30 14:11:19.052010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.051979    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8422e767-014d-4ace-8622-17530ce03eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f9ac612-m2mzc\" (UID: \"8422e767-014d-4ace-8622-17530ce03eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:19.052225 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.052207    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7dtz\" (UniqueName: \"kubernetes.io/projected/8422e767-014d-4ace-8622-17530ce03eee-kube-api-access-v7dtz\") pod \"maas-api-revoke-sub-8f9ac612-m2mzc\" (UID: \"8422e767-014d-4ace-8622-17530ce03eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:19.052275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.052241    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8422e767-014d-4ace-8622-17530ce03eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f9ac612-m2mzc\" (UID: \"8422e767-014d-4ace-8622-17530ce03eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:19.154124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.154093    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7dtz\" (UniqueName: \"kubernetes.io/projected/8422e767-014d-4ace-8622-17530ce03eee-kube-api-access-v7dtz\") pod \"maas-api-revoke-sub-8f9ac612-m2mzc\" (UID: \"8422e767-014d-4ace-8622-17530ce03eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:19.154279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.154139    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8422e767-014d-4ace-8622-17530ce03eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f9ac612-m2mzc\" (UID: \"8422e767-014d-4ace-8622-17530ce03eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:19.154279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.154235    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8422e767-014d-4ace-8622-17530ce03eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f9ac612-m2mzc\" (UID: \"8422e767-014d-4ace-8622-17530ce03eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:19.154926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.154900    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8422e767-014d-4ace-8622-17530ce03eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f9ac612-m2mzc\" (UID: \"8422e767-014d-4ace-8622-17530ce03eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:19.156658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.156615    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8422e767-014d-4ace-8622-17530ce03eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f9ac612-m2mzc\" (UID: \"8422e767-014d-4ace-8622-17530ce03eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:19.180659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.180635    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7dtz\" (UniqueName: \"kubernetes.io/projected/8422e767-014d-4ace-8622-17530ce03eee-kube-api-access-v7dtz\") pod \"maas-api-revoke-sub-8f9ac612-m2mzc\" (UID: \"8422e767-014d-4ace-8622-17530ce03eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:19.264885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.264850    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:19.572473 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:11:19.572439    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8422e767_014d_4ace_8622_17530ce03eee.slice/crio-7d31a6c52b8958a3dd8b78cbce5ca362422ab898082366b9cf7d425c5b444a14 WatchSource:0}: Error finding container 7d31a6c52b8958a3dd8b78cbce5ca362422ab898082366b9cf7d425c5b444a14: Status 404 returned error can't find the container with id 7d31a6c52b8958a3dd8b78cbce5ca362422ab898082366b9cf7d425c5b444a14
Oct 30 14:11:19.576206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.576177    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"]
Oct 30 14:11:19.584842 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.584814    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc" event={"ID":"8422e767-014d-4ace-8622-17530ce03eee","Type":"ContainerStarted","Data":"7d31a6c52b8958a3dd8b78cbce5ca362422ab898082366b9cf7d425c5b444a14"}
Oct 30 14:11:19.586382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.586360    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w" event={"ID":"b9ca9125-32d2-4407-b359-7f3d56602f59","Type":"ContainerDied","Data":"c4a7daf4f34dd0d5556382b3992b9760f324bf381f565f56e0c3f36be5dd8227"}
Oct 30 14:11:19.586453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.586396    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"
Oct 30 14:11:19.586453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:19.586404    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4a7daf4f34dd0d5556382b3992b9760f324bf381f565f56e0c3f36be5dd8227"
Oct 30 14:11:20.596740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:20.596703    4530 generic.go:356] "Generic (PLEG): container finished" podID="8422e767-014d-4ace-8622-17530ce03eee" containerID="cee9b0972ccb7cf02131cec2494d25c3a3572a5aa13f7b78be2a04a6e5df71e9" exitCode=0
Oct 30 14:11:20.597180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:20.596785    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc" event={"ID":"8422e767-014d-4ace-8622-17530ce03eee","Type":"ContainerDied","Data":"cee9b0972ccb7cf02131cec2494d25c3a3572a5aa13f7b78be2a04a6e5df71e9"}
Oct 30 14:11:21.053594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:21.053536    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"]
Oct 30 14:11:21.064374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:21.060552    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37e8566e-7lwtl"]
Oct 30 14:11:21.116250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:21.116218    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8618966-c007-4f7f-ac6c-70e43fb8c614" path="/var/lib/kubelet/pods/e8618966-c007-4f7f-ac6c-70e43fb8c614/volumes"
Oct 30 14:11:21.899422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:21.899393    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:22.009044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.009011    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8422e767-014d-4ace-8622-17530ce03eee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8422e767-014d-4ace-8622-17530ce03eee-maas-api-xks-ca\") pod \"8422e767-014d-4ace-8622-17530ce03eee\" (UID: \"8422e767-014d-4ace-8622-17530ce03eee\") "
Oct 30 14:11:22.009044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.009050    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8422e767-014d-4ace-8622-17530ce03eee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8422e767-014d-4ace-8622-17530ce03eee-maas-api-openshift-ca\") pod \"8422e767-014d-4ace-8622-17530ce03eee\" (UID: \"8422e767-014d-4ace-8622-17530ce03eee\") "
Oct 30 14:11:22.009286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.009079    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8422e767-014d-4ace-8622-17530ce03eee-kube-api-access-v7dtz\" (UniqueName: \"kubernetes.io/projected/8422e767-014d-4ace-8622-17530ce03eee-kube-api-access-v7dtz\") pod \"8422e767-014d-4ace-8622-17530ce03eee\" (UID: \"8422e767-014d-4ace-8622-17530ce03eee\") "
Oct 30 14:11:22.009497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.009466    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8422e767-014d-4ace-8622-17530ce03eee-maas-api-openshift-ca" pod "8422e767-014d-4ace-8622-17530ce03eee" (UID: "8422e767-014d-4ace-8622-17530ce03eee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:22.009937 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.009916    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8422e767-014d-4ace-8622-17530ce03eee-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:22.011087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.011062    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8422e767-014d-4ace-8622-17530ce03eee-maas-api-xks-ca" pod "8422e767-014d-4ace-8622-17530ce03eee" (UID: "8422e767-014d-4ace-8622-17530ce03eee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:22.011232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.011212    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8422e767-014d-4ace-8622-17530ce03eee-kube-api-access-v7dtz" pod "8422e767-014d-4ace-8622-17530ce03eee" (UID: "8422e767-014d-4ace-8622-17530ce03eee"). InnerVolumeSpecName "kube-api-access-v7dtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:22.111314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.111285    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8422e767-014d-4ace-8622-17530ce03eee-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:22.111314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.111314    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7dtz\" (UniqueName: \"kubernetes.io/projected/8422e767-014d-4ace-8622-17530ce03eee-kube-api-access-v7dtz\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:22.614450 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.614422    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"
Oct 30 14:11:22.614636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.614423    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc" event={"ID":"8422e767-014d-4ace-8622-17530ce03eee","Type":"ContainerDied","Data":"7d31a6c52b8958a3dd8b78cbce5ca362422ab898082366b9cf7d425c5b444a14"}
Oct 30 14:11:22.614636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.614575    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d31a6c52b8958a3dd8b78cbce5ca362422ab898082366b9cf7d425c5b444a14"
Oct 30 14:11:22.800713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.800683    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"]
Oct 30 14:11:22.801390 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.801375    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8422e767-014d-4ace-8622-17530ce03eee" containerName="revoke-keys"
Oct 30 14:11:22.801439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.801391    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8422e767-014d-4ace-8622-17530ce03eee" containerName="revoke-keys"
Oct 30 14:11:22.801511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.801499    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8422e767-014d-4ace-8622-17530ce03eee" containerName="revoke-keys"
Oct 30 14:11:22.805373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.805355    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:22.824209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.824183    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:11:22.824371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.824299    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:11:22.824908 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.824867    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"]
Oct 30 14:11:22.921985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.921911    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-476cdbdd-6hhhw\" (UID: \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:22.921985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.921952    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx5tm\" (UniqueName: \"kubernetes.io/projected/b2bef961-40fb-4eb6-9d68-faf4caa9faee-kube-api-access-sx5tm\") pod \"maas-api-revoke-sub-476cdbdd-6hhhw\" (UID: \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:22.922346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:22.922148    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-476cdbdd-6hhhw\" (UID: \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:23.023673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:23.023639    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-476cdbdd-6hhhw\" (UID: \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:23.023848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:23.023689    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx5tm\" (UniqueName: \"kubernetes.io/projected/b2bef961-40fb-4eb6-9d68-faf4caa9faee-kube-api-access-sx5tm\") pod \"maas-api-revoke-sub-476cdbdd-6hhhw\" (UID: \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:23.023921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:23.023881    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-476cdbdd-6hhhw\" (UID: \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:23.024362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:23.024333    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-476cdbdd-6hhhw\" (UID: \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:23.026110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:23.026090    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-476cdbdd-6hhhw\" (UID: \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:23.052616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:23.052596    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx5tm\" (UniqueName: \"kubernetes.io/projected/b2bef961-40fb-4eb6-9d68-faf4caa9faee-kube-api-access-sx5tm\") pod \"maas-api-revoke-sub-476cdbdd-6hhhw\" (UID: \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:23.124248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:23.124221    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:23.431328 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:11:23.431294    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bef961_40fb_4eb6_9d68_faf4caa9faee.slice/crio-af2c73be6aad345a669d1273673b886001e150dd83cba8af552dc0e873ebc0df WatchSource:0}: Error finding container af2c73be6aad345a669d1273673b886001e150dd83cba8af552dc0e873ebc0df: Status 404 returned error can't find the container with id af2c73be6aad345a669d1273673b886001e150dd83cba8af552dc0e873ebc0df
Oct 30 14:11:23.434443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:23.434420    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"]
Oct 30 14:11:23.634991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:23.634954    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw" event={"ID":"b2bef961-40fb-4eb6-9d68-faf4caa9faee","Type":"ContainerStarted","Data":"af2c73be6aad345a669d1273673b886001e150dd83cba8af552dc0e873ebc0df"}
Oct 30 14:11:24.644622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:24.644580    4530 generic.go:356] "Generic (PLEG): container finished" podID="b2bef961-40fb-4eb6-9d68-faf4caa9faee" containerID="f2eed58729d85b9dc97c463b0f43793b8d988306b80c7883340a525d5a21b526" exitCode=0
Oct 30 14:11:24.644995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:24.644661    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw" event={"ID":"b2bef961-40fb-4eb6-9d68-faf4caa9faee","Type":"ContainerDied","Data":"f2eed58729d85b9dc97c463b0f43793b8d988306b80c7883340a525d5a21b526"}
Oct 30 14:11:25.034877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.034791    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"]
Oct 30 14:11:25.039258 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.039240    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.076641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.076610    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"]
Oct 30 14:11:25.163855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.163824    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.164015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.163868    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.164061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.164008    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.164139 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.164119    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.164200 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.164189    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58017dd-1638-4df7-a7f9-dffd6db076f6-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.164315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.164300    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.164362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.164350    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.164397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.164376    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vt7p\" (UniqueName: \"kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-kube-api-access-9vt7p\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.164470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.164457    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.275566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.266697    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58017dd-1638-4df7-a7f9-dffd6db076f6-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.305632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.303313    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.305632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.303397    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.305632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.303427    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vt7p\" (UniqueName: \"kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-kube-api-access-9vt7p\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.305632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.303497    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.305632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.303614    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.305632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.303682    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.305632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.303869    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.305632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.303991    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.305632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.304556    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.305632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.301912    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58017dd-1638-4df7-a7f9-dffd6db076f6-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.305632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.304855    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.308374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.308257    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.322659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.321799    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.350563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.343968    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.351151 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.351085    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.377565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.370196    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.377565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.376709    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vt7p\" (UniqueName: \"kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-kube-api-access-9vt7p\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.653213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.653131    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:25.826990 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.826964    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:25.913662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.913596    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-openshift-ca\") pod \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\" (UID: \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\") "
Oct 30 14:11:25.913955 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.913932    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-xks-ca\") pod \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\" (UID: \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\") "
Oct 30 14:11:25.914011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.913959    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-openshift-ca" pod "b2bef961-40fb-4eb6-9d68-faf4caa9faee" (UID: "b2bef961-40fb-4eb6-9d68-faf4caa9faee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:25.914011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.913994    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bef961-40fb-4eb6-9d68-faf4caa9faee-kube-api-access-sx5tm\" (UniqueName: \"kubernetes.io/projected/b2bef961-40fb-4eb6-9d68-faf4caa9faee-kube-api-access-sx5tm\") pod \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\" (UID: \"b2bef961-40fb-4eb6-9d68-faf4caa9faee\") "
Oct 30 14:11:25.914610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.914584    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:25.916550 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.916506    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-xks-ca" pod "b2bef961-40fb-4eb6-9d68-faf4caa9faee" (UID: "b2bef961-40fb-4eb6-9d68-faf4caa9faee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:25.916670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.916577    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bef961-40fb-4eb6-9d68-faf4caa9faee-kube-api-access-sx5tm" pod "b2bef961-40fb-4eb6-9d68-faf4caa9faee" (UID: "b2bef961-40fb-4eb6-9d68-faf4caa9faee"). InnerVolumeSpecName "kube-api-access-sx5tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:25.979068 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:11:25.979041    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58017dd_1638_4df7_a7f9_dffd6db076f6.slice/crio-f4638efc3b354dcef9b753f505b7f0576c97cd8f2e76a41000c914d703d20739 WatchSource:0}: Error finding container f4638efc3b354dcef9b753f505b7f0576c97cd8f2e76a41000c914d703d20739: Status 404 returned error can't find the container with id f4638efc3b354dcef9b753f505b7f0576c97cd8f2e76a41000c914d703d20739
Oct 30 14:11:25.981767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.981732    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:11:25.981848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.981800    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:11:25.981848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.981829    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:11:25.982788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:25.982767    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"]
Oct 30 14:11:26.016039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:26.016012    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bef961-40fb-4eb6-9d68-faf4caa9faee-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:26.016039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:26.016041    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx5tm\" (UniqueName: \"kubernetes.io/projected/b2bef961-40fb-4eb6-9d68-faf4caa9faee-kube-api-access-sx5tm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:26.619183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:26.619095    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:11:26.663226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:26.663195    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"
Oct 30 14:11:26.663626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:26.663242    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw" event={"ID":"b2bef961-40fb-4eb6-9d68-faf4caa9faee","Type":"ContainerDied","Data":"af2c73be6aad345a669d1273673b886001e150dd83cba8af552dc0e873ebc0df"}
Oct 30 14:11:26.663626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:26.663298    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af2c73be6aad345a669d1273673b886001e150dd83cba8af552dc0e873ebc0df"
Oct 30 14:11:26.665225 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:26.665198    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" event={"ID":"e58017dd-1638-4df7-a7f9-dffd6db076f6","Type":"ContainerStarted","Data":"69262667ecdf8d545bdb6167e6cb61256031486673cff2095bc82dedf9b9b17a"}
Oct 30 14:11:26.665358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:26.665240    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" event={"ID":"e58017dd-1638-4df7-a7f9-dffd6db076f6","Type":"ContainerStarted","Data":"f4638efc3b354dcef9b753f505b7f0576c97cd8f2e76a41000c914d703d20739"}
Oct 30 14:11:26.726767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:26.726704    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podStartSLOduration=1.726684964 podStartE2EDuration="1.726684964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:26.720373926 +0000 UTC m=+1351.752272877" watchObservedRunningTime="2026-10-30 14:11:26.726684964 +0000 UTC m=+1351.758583907"
Oct 30 14:11:27.657567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:27.656870    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:27.661560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:27.658020    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:27.661560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:27.658082    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:28.661568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:28.659007    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:28.661568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:28.659070    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:29.657020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:29.656981    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:29.657297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:29.657269    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:30.059563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:30.057460    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:11:30.100564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:30.099640    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:30.145565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:30.144629    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:30.467741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:30.467630    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:30.663592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:30.654940    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:30.663592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:30.655003    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:31.114507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:31.114452    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:11:31.153137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:31.140746    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:31.174922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:31.174785    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:31.233627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:31.178087    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:11:31.233627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:31.184693    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:11:31.233627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:31.184736    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:11:31.702626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:31.562690    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:31.702626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:31.573735    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:11:31.993631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:31.993196    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:31.993631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:31.993290    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:32.130574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.122719    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:32.130574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.123771    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": context deadline exceeded"
Oct 30 14:11:32.256629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.211987    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:32.263144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.234219    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 14:11:32.475120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.475068    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:32.475440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.475409    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" podUID="259f2601-7ea2-43b0-abeb-73891558a746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:32.671566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.663647    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:32.761566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.754721    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Oct 30 14:11:32.779570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.771635    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:32.811896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.807469    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Oct 30 14:11:32.823465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.701595    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:32.823465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.819315    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" podUID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:33.023626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.998909    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:33.023626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:32.998974    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:33.134362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:33.123132    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:33.278817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:33.276804    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:33.294668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:33.294603    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:33.308712 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:33.308674    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:33.309002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:33.308965    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" podUID="cb30f186-609d-4e05-9018-9ce4b63a133c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:33.660647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:33.659986    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:33.660647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:33.660065    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:33.763921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:33.762646    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:33.832410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:33.832107    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:33.832410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:33.832164    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" podUID="0e166404-62ee-4c62-83b7-ba6375714a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:33.903122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:33.903080    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:33.912750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:33.912655    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:34.064343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.064308    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:34.064623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.064592    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:34.221404 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.221304    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"]
Oct 30 14:11:34.230724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.228471    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:34.289010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.288965    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:34.309568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.299824    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:34.311186 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.311152    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:34.311410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.311382    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:34.311624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.311605    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:34.311754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.311735    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:34.553234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.553185    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557aac5b-xw9k4"]
Oct 30 14:11:34.656350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.654777    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:34.656350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.654837    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:34.729584 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.727678    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:34.729584 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.727745    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:34.768642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:34.767769    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:35.146025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:35.145986    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:35.146372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:35.146326    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:35.287581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:35.278724    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:35.304684 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:35.304638    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:35.305627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:35.305593    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:35.710183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:35.710139    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:35.710492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:35.710457    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:35.844619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:35.832731    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Oct 30 14:11:36.243566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.198917    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147baac4-8f45-4a17-9728-1d3483a300f3" path="/var/lib/kubelet/pods/147baac4-8f45-4a17-9728-1d3483a300f3/volumes"
Oct 30 14:11:36.464230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.464182    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded"
Oct 30 14:11:36.468883 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:11:36.468842    4530 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.318s"
Oct 30 14:11:36.469108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.469089    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"]
Oct 30 14:11:36.682061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.681514    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b2bef961-40fb-4eb6-9d68-faf4caa9faee" containerName="revoke-keys"
Oct 30 14:11:36.682061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.681567    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bef961-40fb-4eb6-9d68-faf4caa9faee" containerName="revoke-keys"
Oct 30 14:11:36.722462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.722420    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bef961-40fb-4eb6-9d68-faf4caa9faee" containerName="revoke-keys"
Oct 30 14:11:36.850213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.726061    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:11:36.850213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.726093    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 14:11:36.850213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.769945    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:36.850213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.847059    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:36.905578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.854951    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"]
Oct 30 14:11:36.905578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.856086    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:36.905578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.858136    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:36.905578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.886110    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:11:36.905578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.894259    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:11:36.905578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.902044    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:11:36.965624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.964643    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktx46\" (UniqueName: \"kubernetes.io/projected/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-kube-api-access-ktx46\") pod \"maas-api-revoke-sub-68f39a02-xlg8b\" (UID: \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:36.965624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.964748    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68f39a02-xlg8b\" (UID: \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:36.965624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:36.964822    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68f39a02-xlg8b\" (UID: \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:37.074046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.074000    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktx46\" (UniqueName: \"kubernetes.io/projected/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-kube-api-access-ktx46\") pod \"maas-api-revoke-sub-68f39a02-xlg8b\" (UID: \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:37.075524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.075494    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68f39a02-xlg8b\" (UID: \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:37.204108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.204074    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68f39a02-xlg8b\" (UID: \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:37.205354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.205156    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68f39a02-xlg8b\" (UID: \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:37.286120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.268347    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktx46\" (UniqueName: \"kubernetes.io/projected/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-kube-api-access-ktx46\") pod \"maas-api-revoke-sub-68f39a02-xlg8b\" (UID: \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:37.287223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.287191    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68f39a02-xlg8b\" (UID: \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:37.300163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.299670    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:37.300163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.299733    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:37.300691 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.300395    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:37.320809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.312334    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:11:37.320809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.312392    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:37.390121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.390086    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:11:37.472571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.471666    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:37.658323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.658289    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:37.658525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.658501    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:37.869098 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:37.866186    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:38.096817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:38.096763    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:38.222032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:38.220653    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:38.318613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:38.253904    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:11:38.472753 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:38.472334    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:11:38.653437 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:11:38.653404    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74cb90f_b27c_49fd_b8eb_39dfaeaacb4d.slice/crio-67d3a7366a6535222140dc965fbad9b8f04983a6110a4de89e62d24a237a2b62 WatchSource:0}: Error finding container 67d3a7366a6535222140dc965fbad9b8f04983a6110a4de89e62d24a237a2b62: Status 404 returned error can't find the container with id 67d3a7366a6535222140dc965fbad9b8f04983a6110a4de89e62d24a237a2b62
Oct 30 14:11:38.654697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:38.654670    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:38.654826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:38.654722    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:38.655695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:38.655674    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"]
Oct 30 14:11:38.916268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:38.916187    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b" event={"ID":"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d","Type":"ContainerStarted","Data":"67d3a7366a6535222140dc965fbad9b8f04983a6110a4de89e62d24a237a2b62"}
Oct 30 14:11:39.654338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:39.654301    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:39.654842 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:39.654359    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:39.927822 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:39.927733    4530 generic.go:356] "Generic (PLEG): container finished" podID="b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d" containerID="751212252ddc65e1f9be25a08c6f6ae64e45fd827c78cd3be0fd6422588549b7" exitCode=0
Oct 30 14:11:39.927972 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:39.927815    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b" event={"ID":"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d","Type":"ContainerDied","Data":"751212252ddc65e1f9be25a08c6f6ae64e45fd827c78cd3be0fd6422588549b7"}
Oct 30 14:11:40.150602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.146166    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9591-85f6f7"]
Oct 30 14:11:40.171609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.169728    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9591-85f6f7"]
Oct 30 14:11:40.171609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.169852    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-85f6f7"
Oct 30 14:11:40.201404 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.199004    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q95l7\""
Oct 30 14:11:40.223304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.220503    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6rrv\" (UniqueName: \"kubernetes.io/projected/47ec24d4-a985-4b4f-8143-1c25ec25783c-kube-api-access-b6rrv\") pod \"test-curl-9591-85f6f7\" (UID: \"47ec24d4-a985-4b4f-8143-1c25ec25783c\") " pod="opendatahub/test-curl-9591-85f6f7"
Oct 30 14:11:40.322288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.322250    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6rrv\" (UniqueName: \"kubernetes.io/projected/47ec24d4-a985-4b4f-8143-1c25ec25783c-kube-api-access-b6rrv\") pod \"test-curl-9591-85f6f7\" (UID: \"47ec24d4-a985-4b4f-8143-1c25ec25783c\") " pod="opendatahub/test-curl-9591-85f6f7"
Oct 30 14:11:40.350368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.350338    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6rrv\" (UniqueName: \"kubernetes.io/projected/47ec24d4-a985-4b4f-8143-1c25ec25783c-kube-api-access-b6rrv\") pod \"test-curl-9591-85f6f7\" (UID: \"47ec24d4-a985-4b4f-8143-1c25ec25783c\") " pod="opendatahub/test-curl-9591-85f6f7"
Oct 30 14:11:40.499285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.499200    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-85f6f7"
Oct 30 14:11:40.654101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.654070    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:40.654408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.654383    4530 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 14:11:40.654782 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.654441    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 14:11:40.810957 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:11:40.810929    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47ec24d4_a985_4b4f_8143_1c25ec25783c.slice/crio-b9f6b71b9e2bc6dee7d6fa1a1ed380a4d8250023cb99b833661666a4a3430c7a WatchSource:0}: Error finding container b9f6b71b9e2bc6dee7d6fa1a1ed380a4d8250023cb99b833661666a4a3430c7a: Status 404 returned error can't find the container with id b9f6b71b9e2bc6dee7d6fa1a1ed380a4d8250023cb99b833661666a4a3430c7a
Oct 30 14:11:40.813219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.813198    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:11:40.816381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.816355    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9591-85f6f7"]
Oct 30 14:11:40.939377 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:40.939336    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9591-85f6f7" event={"ID":"47ec24d4-a985-4b4f-8143-1c25ec25783c","Type":"ContainerStarted","Data":"b9f6b71b9e2bc6dee7d6fa1a1ed380a4d8250023cb99b833661666a4a3430c7a"}
Oct 30 14:11:41.146669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.146642    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:41.236283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.236246    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-xks-ca\") pod \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\" (UID: \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\") "
Oct 30 14:11:41.236467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.236421    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-kube-api-access-ktx46\" (UniqueName: \"kubernetes.io/projected/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-kube-api-access-ktx46\") pod \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\" (UID: \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\") "
Oct 30 14:11:41.236807 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.236784    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-openshift-ca\") pod \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\" (UID: \"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d\") "
Oct 30 14:11:41.237126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.237093    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-openshift-ca" pod "b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d" (UID: "b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:41.237635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.237613    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:41.239518 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.239489    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-xks-ca" pod "b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d" (UID: "b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:41.240336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.240312    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-kube-api-access-ktx46" pod "b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d" (UID: "b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d"). InnerVolumeSpecName "kube-api-access-ktx46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:41.339000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.338974    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:41.339000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.338999    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktx46\" (UniqueName: \"kubernetes.io/projected/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d-kube-api-access-ktx46\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:41.372696 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:11:41.372572    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:11:41.659112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.659034    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:41.660177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.660155    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:11:41.951498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.951420    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"
Oct 30 14:11:41.951498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.951434    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b" event={"ID":"b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d","Type":"ContainerDied","Data":"67d3a7366a6535222140dc965fbad9b8f04983a6110a4de89e62d24a237a2b62"}
Oct 30 14:11:41.951498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:41.951489    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67d3a7366a6535222140dc965fbad9b8f04983a6110a4de89e62d24a237a2b62"
Oct 30 14:11:43.997409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:43.997365    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9591-85f6f7" event={"ID":"47ec24d4-a985-4b4f-8143-1c25ec25783c","Type":"ContainerStarted","Data":"d122b3be604395b82560b6a951232c6020df5fa332268080dd0c63d239781c6f"}
Oct 30 14:11:44.054989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:44.054938    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9591-85f6f7" podStartSLOduration=1.783606148 podStartE2EDuration="4.054921661s" totalImagesPullingTime="2.271315513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:44.051682093 +0000 UTC m=+1369.083581034" watchObservedRunningTime="2026-10-30 14:11:44.054921661 +0000 UTC m=+1369.086820599"
Oct 30 14:11:44.970814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:44.970778    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9591-85f6f7"]
Oct 30 14:11:44.977240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:44.977207    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9591-85f6f7"]
Oct 30 14:11:45.137151 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.137119    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9591-55517f"]
Oct 30 14:11:45.137805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.137790    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d" containerName="revoke-keys"
Oct 30 14:11:45.137853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.137806    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d" containerName="revoke-keys"
Oct 30 14:11:45.137853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.137822    4530 cpu_manager.go:414] "containerMap: removing container" podUID="47ec24d4-a985-4b4f-8143-1c25ec25783c" containerName="test-curl-9591-85f6f7"
Oct 30 14:11:45.137853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.137828    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ec24d4-a985-4b4f-8143-1c25ec25783c" containerName="test-curl-9591-85f6f7"
Oct 30 14:11:45.137958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.137948    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d" containerName="revoke-keys"
Oct 30 14:11:45.137998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.137961    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ec24d4-a985-4b4f-8143-1c25ec25783c" containerName="test-curl-9591-85f6f7"
Oct 30 14:11:45.141942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.141925    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-55517f"
Oct 30 14:11:45.158233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.158205    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9591-55517f"]
Oct 30 14:11:45.203768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.203735    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjps7\" (UniqueName: \"kubernetes.io/projected/5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b-kube-api-access-sjps7\") pod \"test-curl-9591-55517f\" (UID: \"5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b\") " pod="opendatahub/test-curl-9591-55517f"
Oct 30 14:11:45.305094 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.305063    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjps7\" (UniqueName: \"kubernetes.io/projected/5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b-kube-api-access-sjps7\") pod \"test-curl-9591-55517f\" (UID: \"5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b\") " pod="opendatahub/test-curl-9591-55517f"
Oct 30 14:11:45.332328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.332302    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjps7\" (UniqueName: \"kubernetes.io/projected/5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b-kube-api-access-sjps7\") pod \"test-curl-9591-55517f\" (UID: \"5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b\") " pod="opendatahub/test-curl-9591-55517f"
Oct 30 14:11:45.457230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.457203    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-55517f"
Oct 30 14:11:45.764733 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:11:45.764699    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ae50602_eac0_4cc9_b20e_b1ffc7f9d77b.slice/crio-5233e70cde866c262df72d3a6fc2dcb850ae3ad87ba525abd111ba536684798d WatchSource:0}: Error finding container 5233e70cde866c262df72d3a6fc2dcb850ae3ad87ba525abd111ba536684798d: Status 404 returned error can't find the container with id 5233e70cde866c262df72d3a6fc2dcb850ae3ad87ba525abd111ba536684798d
Oct 30 14:11:45.767189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:45.767167    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9591-55517f"]
Oct 30 14:11:46.039788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:46.031949    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9591-85f6f7" podUID="47ec24d4-a985-4b4f-8143-1c25ec25783c" containerName="test-curl-9591-85f6f7" containerID="cri-o://d122b3be604395b82560b6a951232c6020df5fa332268080dd0c63d239781c6f" gracePeriod=2
Oct 30 14:11:46.043649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:46.039980    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9591-55517f" event={"ID":"5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b","Type":"ContainerStarted","Data":"5151904dd6e9a412519eeb6379437dc491353d27c65d495f3cc46024ddfd0247"}
Oct 30 14:11:46.043649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:46.040033    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9591-55517f" event={"ID":"5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b","Type":"ContainerStarted","Data":"5233e70cde866c262df72d3a6fc2dcb850ae3ad87ba525abd111ba536684798d"}
Oct 30 14:11:46.091913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:46.091857    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9591-55517f" podStartSLOduration=1.091841246 podStartE2EDuration="1.091841246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:46.086063698 +0000 UTC m=+1371.117962637" watchObservedRunningTime="2026-10-30 14:11:46.091841246 +0000 UTC m=+1371.123740185"
Oct 30 14:11:47.057252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:47.057214    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9591-55517f"]
Oct 30 14:11:47.065302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:47.065205    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9591-55517f"]
Oct 30 14:11:48.043929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.043896    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"]
Oct 30 14:11:48.049171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.049123    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9591-55517f" podUID="5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b" containerName="test-curl-9591-55517f" containerID="cri-o://5151904dd6e9a412519eeb6379437dc491353d27c65d495f3cc46024ddfd0247" gracePeriod=2
Oct 30 14:11:48.058683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.058648    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c576cb31-fjhds"]
Oct 30 14:11:48.241827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.241802    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-85f6f7"
Oct 30 14:11:48.356467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.356442    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ec24d4-a985-4b4f-8143-1c25ec25783c-kube-api-access-b6rrv\" (UniqueName: \"kubernetes.io/projected/47ec24d4-a985-4b4f-8143-1c25ec25783c-kube-api-access-b6rrv\") pod \"47ec24d4-a985-4b4f-8143-1c25ec25783c\" (UID: \"47ec24d4-a985-4b4f-8143-1c25ec25783c\") "
Oct 30 14:11:48.358510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.358483    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ec24d4-a985-4b4f-8143-1c25ec25783c-kube-api-access-b6rrv" pod "47ec24d4-a985-4b4f-8143-1c25ec25783c" (UID: "47ec24d4-a985-4b4f-8143-1c25ec25783c"). InnerVolumeSpecName "kube-api-access-b6rrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:48.458160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.458119    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6rrv\" (UniqueName: \"kubernetes.io/projected/47ec24d4-a985-4b4f-8143-1c25ec25783c-kube-api-access-b6rrv\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:48.527245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.527218    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"]
Oct 30 14:11:48.527904 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.527890    4530 cpu_manager.go:414] "containerMap: removing container" podUID="5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b" containerName="test-curl-9591-55517f"
Oct 30 14:11:48.527955 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.527906    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b" containerName="test-curl-9591-55517f"
Oct 30 14:11:48.528046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.528037    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b" containerName="test-curl-9591-55517f"
Oct 30 14:11:48.531842 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.531827    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:48.550868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.550841    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"]
Oct 30 14:11:48.558799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.558756    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:11:48.558799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.558789    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:11:48.660235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.660170    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ae48173-2zqw2\" (UID: \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:48.660353 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.660329    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9spcz\" (UniqueName: \"kubernetes.io/projected/0ea53d4d-91a5-44d3-9b42-e291de0856f7-kube-api-access-9spcz\") pod \"maas-api-revoke-sub-1ae48173-2zqw2\" (UID: \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:48.660519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.660501    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ae48173-2zqw2\" (UID: \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:48.762786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.762683    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ae48173-2zqw2\" (UID: \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:48.762936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.762906    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ae48173-2zqw2\" (UID: \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:48.763088 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.763072    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9spcz\" (UniqueName: \"kubernetes.io/projected/0ea53d4d-91a5-44d3-9b42-e291de0856f7-kube-api-access-9spcz\") pod \"maas-api-revoke-sub-1ae48173-2zqw2\" (UID: \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:48.763688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.763665    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ae48173-2zqw2\" (UID: \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:48.765327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.765306    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ae48173-2zqw2\" (UID: \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:48.790463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.790440    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9spcz\" (UniqueName: \"kubernetes.io/projected/0ea53d4d-91a5-44d3-9b42-e291de0856f7-kube-api-access-9spcz\") pod \"maas-api-revoke-sub-1ae48173-2zqw2\" (UID: \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:48.846519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:48.846499    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:49.060886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:49.060844    4530 generic.go:356] "Generic (PLEG): container finished" podID="47ec24d4-a985-4b4f-8143-1c25ec25783c" containerID="d122b3be604395b82560b6a951232c6020df5fa332268080dd0c63d239781c6f" exitCode=137
Oct 30 14:11:49.061315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:49.060905    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-85f6f7"
Oct 30 14:11:49.061315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:49.060943    4530 scope.go:122] "RemoveContainer" containerID="d122b3be604395b82560b6a951232c6020df5fa332268080dd0c63d239781c6f"
Oct 30 14:11:49.080694 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:49.080673    4530 scope.go:122] "RemoveContainer" containerID="d122b3be604395b82560b6a951232c6020df5fa332268080dd0c63d239781c6f"
Oct 30 14:11:49.080955 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:11:49.080937    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d122b3be604395b82560b6a951232c6020df5fa332268080dd0c63d239781c6f\": container with ID starting with d122b3be604395b82560b6a951232c6020df5fa332268080dd0c63d239781c6f not found: ID does not exist" containerID="d122b3be604395b82560b6a951232c6020df5fa332268080dd0c63d239781c6f"
Oct 30 14:11:49.081008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:49.080963    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d122b3be604395b82560b6a951232c6020df5fa332268080dd0c63d239781c6f"} err="failed to get container status \"d122b3be604395b82560b6a951232c6020df5fa332268080dd0c63d239781c6f\": rpc error: code = NotFound desc = could not find container \"d122b3be604395b82560b6a951232c6020df5fa332268080dd0c63d239781c6f\": container with ID starting with d122b3be604395b82560b6a951232c6020df5fa332268080dd0c63d239781c6f not found: ID does not exist"
Oct 30 14:11:49.091686 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:11:49.091660    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9591-85f6f7\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="47ec24d4-a985-4b4f-8143-1c25ec25783c" pod="opendatahub/test-curl-9591-85f6f7"
Oct 30 14:11:49.117472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:49.117447    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd19176-0f16-4091-a38f-4b51aaaa80c2" path="/var/lib/kubelet/pods/0dd19176-0f16-4091-a38f-4b51aaaa80c2/volumes"
Oct 30 14:11:49.118000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:49.117977    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ec24d4-a985-4b4f-8143-1c25ec25783c" path="/var/lib/kubelet/pods/47ec24d4-a985-4b4f-8143-1c25ec25783c/volumes"
Oct 30 14:11:49.159696 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:11:49.159667    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea53d4d_91a5_44d3_9b42_e291de0856f7.slice/crio-9b4bef9bf1c0dde97f6a048f4f928427dacdb460417ea34f09d7773886bfbfba WatchSource:0}: Error finding container 9b4bef9bf1c0dde97f6a048f4f928427dacdb460417ea34f09d7773886bfbfba: Status 404 returned error can't find the container with id 9b4bef9bf1c0dde97f6a048f4f928427dacdb460417ea34f09d7773886bfbfba
Oct 30 14:11:49.161867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:49.161765    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"]
Oct 30 14:11:50.072411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:50.072370    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2" event={"ID":"0ea53d4d-91a5-44d3-9b42-e291de0856f7","Type":"ContainerStarted","Data":"745c0a9fb0c3a5f9854ace05aea34e9cf8caa9c219820e9189adc0764354285d"}
Oct 30 14:11:50.072797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:50.072475    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2" event={"ID":"0ea53d4d-91a5-44d3-9b42-e291de0856f7","Type":"ContainerStarted","Data":"9b4bef9bf1c0dde97f6a048f4f928427dacdb460417ea34f09d7773886bfbfba"}
Oct 30 14:11:50.122954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:50.122887    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2" podStartSLOduration=2.122867405 podStartE2EDuration="2.122867405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:50.117612026 +0000 UTC m=+1375.149510964" watchObservedRunningTime="2026-10-30 14:11:50.122867405 +0000 UTC m=+1375.154766344"
Oct 30 14:11:50.282137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:50.282113    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-55517f"
Oct 30 14:11:50.401592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:50.401518    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b-kube-api-access-sjps7\" (UniqueName: \"kubernetes.io/projected/5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b-kube-api-access-sjps7\") pod \"5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b\" (UID: \"5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b\") "
Oct 30 14:11:50.403528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:50.403500    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b-kube-api-access-sjps7" pod "5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b" (UID: "5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b"). InnerVolumeSpecName "kube-api-access-sjps7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:50.503635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:50.503606    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjps7\" (UniqueName: \"kubernetes.io/projected/5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b-kube-api-access-sjps7\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.082203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:51.082164    4530 generic.go:356] "Generic (PLEG): container finished" podID="0ea53d4d-91a5-44d3-9b42-e291de0856f7" containerID="745c0a9fb0c3a5f9854ace05aea34e9cf8caa9c219820e9189adc0764354285d" exitCode=0
Oct 30 14:11:51.082619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:51.082207    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2" event={"ID":"0ea53d4d-91a5-44d3-9b42-e291de0856f7","Type":"ContainerDied","Data":"745c0a9fb0c3a5f9854ace05aea34e9cf8caa9c219820e9189adc0764354285d"}
Oct 30 14:11:51.083887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:51.083861    4530 generic.go:356] "Generic (PLEG): container finished" podID="5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b" containerID="5151904dd6e9a412519eeb6379437dc491353d27c65d495f3cc46024ddfd0247" exitCode=137
Oct 30 14:11:51.084021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:51.083897    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-55517f"
Oct 30 14:11:51.084021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:51.083928    4530 scope.go:122] "RemoveContainer" containerID="5151904dd6e9a412519eeb6379437dc491353d27c65d495f3cc46024ddfd0247"
Oct 30 14:11:51.094836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:51.094687    4530 scope.go:122] "RemoveContainer" containerID="5151904dd6e9a412519eeb6379437dc491353d27c65d495f3cc46024ddfd0247"
Oct 30 14:11:51.094984 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:11:51.094964    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5151904dd6e9a412519eeb6379437dc491353d27c65d495f3cc46024ddfd0247\": container with ID starting with 5151904dd6e9a412519eeb6379437dc491353d27c65d495f3cc46024ddfd0247 not found: ID does not exist" containerID="5151904dd6e9a412519eeb6379437dc491353d27c65d495f3cc46024ddfd0247"
Oct 30 14:11:51.095035 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:51.094994    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5151904dd6e9a412519eeb6379437dc491353d27c65d495f3cc46024ddfd0247"} err="failed to get container status \"5151904dd6e9a412519eeb6379437dc491353d27c65d495f3cc46024ddfd0247\": rpc error: code = NotFound desc = could not find container \"5151904dd6e9a412519eeb6379437dc491353d27c65d495f3cc46024ddfd0247\": container with ID starting with 5151904dd6e9a412519eeb6379437dc491353d27c65d495f3cc46024ddfd0247 not found: ID does not exist"
Oct 30 14:11:51.116569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:51.116528    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b" path="/var/lib/kubelet/pods/5ae50602-eac0-4cc9-b20e-b1ffc7f9d77b/volumes"
Oct 30 14:11:52.279508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:52.279482    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:52.435118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:52.435030    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-xks-ca\") pod \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\" (UID: \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\") "
Oct 30 14:11:52.435283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:52.435131    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-openshift-ca\") pod \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\" (UID: \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\") "
Oct 30 14:11:52.435369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:52.435345    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea53d4d-91a5-44d3-9b42-e291de0856f7-kube-api-access-9spcz\" (UniqueName: \"kubernetes.io/projected/0ea53d4d-91a5-44d3-9b42-e291de0856f7-kube-api-access-9spcz\") pod \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\" (UID: \"0ea53d4d-91a5-44d3-9b42-e291de0856f7\") "
Oct 30 14:11:52.435527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:52.435500    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-openshift-ca" pod "0ea53d4d-91a5-44d3-9b42-e291de0856f7" (UID: "0ea53d4d-91a5-44d3-9b42-e291de0856f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:52.435951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:52.435929    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:52.437199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:52.437172    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-xks-ca" pod "0ea53d4d-91a5-44d3-9b42-e291de0856f7" (UID: "0ea53d4d-91a5-44d3-9b42-e291de0856f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:52.437397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:52.437382    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea53d4d-91a5-44d3-9b42-e291de0856f7-kube-api-access-9spcz" pod "0ea53d4d-91a5-44d3-9b42-e291de0856f7" (UID: "0ea53d4d-91a5-44d3-9b42-e291de0856f7"). InnerVolumeSpecName "kube-api-access-9spcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:52.536977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:52.536947    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9spcz\" (UniqueName: \"kubernetes.io/projected/0ea53d4d-91a5-44d3-9b42-e291de0856f7-kube-api-access-9spcz\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:52.536977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:52.536975    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea53d4d-91a5-44d3-9b42-e291de0856f7-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:11:53.113632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:53.113603    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"
Oct 30 14:11:53.116368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:53.116344    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2" event={"ID":"0ea53d4d-91a5-44d3-9b42-e291de0856f7","Type":"ContainerDied","Data":"9b4bef9bf1c0dde97f6a048f4f928427dacdb460417ea34f09d7773886bfbfba"}
Oct 30 14:11:53.116484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:53.116382    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b4bef9bf1c0dde97f6a048f4f928427dacdb460417ea34f09d7773886bfbfba"
Oct 30 14:11:55.119668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.119637    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"]
Oct 30 14:11:55.120357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.120340    4530 cpu_manager.go:414] "containerMap: removing container" podUID="0ea53d4d-91a5-44d3-9b42-e291de0856f7" containerName="revoke-keys"
Oct 30 14:11:55.120458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.120359    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea53d4d-91a5-44d3-9b42-e291de0856f7" containerName="revoke-keys"
Oct 30 14:11:55.120492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.120483    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea53d4d-91a5-44d3-9b42-e291de0856f7" containerName="revoke-keys"
Oct 30 14:11:55.124723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.124699    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.145468 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.145407    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"]
Oct 30 14:11:55.325765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.325715    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.326192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.326162    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.326533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.326511    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwz2\" (UniqueName: \"kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-kube-api-access-cnwz2\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.326731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.326710    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.326914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.326895    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.327378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.327311    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b8b223-4999-4a21-a275-093515f6ccfe-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.327620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.327600    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.327824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.327808    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b8b223-4999-4a21-a275-093515f6ccfe-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.328037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.328017    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.431746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.431648    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.432558 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.432514    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.432792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.432773    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwz2\" (UniqueName: \"kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-kube-api-access-cnwz2\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.432915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.432900    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.433061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.433047    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.444565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.433187    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b8b223-4999-4a21-a275-093515f6ccfe-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.444810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.444786    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.445075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.445056    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b8b223-4999-4a21-a275-093515f6ccfe-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.445194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.445180    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.456564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.445765    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.456800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.440747    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.456917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.451524    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b8b223-4999-4a21-a275-093515f6ccfe-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.457062 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.451783    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.459715 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.459677    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.478563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.462680    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b8b223-4999-4a21-a275-093515f6ccfe-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.497577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.497523    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwz2\" (UniqueName: \"kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-kube-api-access-cnwz2\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.500866 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.500835    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.501851 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.501807    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:55.741283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:55.740800    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:11:56.950068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:56.936674    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:57.130569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:57.129627    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:11:57.380255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:57.358586    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 14:11:57.381721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:57.366911    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:57.458038 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:57.455348    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"]
Oct 30 14:11:57.524582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:57.517830    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:11:57.524582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:57.517930    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:11:57.524582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:57.517978    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:11:57.938845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:57.938802    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:57.940323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:57.940302    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:11:58.092917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:58.092859    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:58.130018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:58.129972    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:58.131157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:58.131130    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:11:58.232952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:58.163827    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:58.381569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:58.380627    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:58.381569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:58.381053    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:11:58.395611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:58.382093    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:58.395611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:58.382428    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:11:58.692455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:58.628085    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"]
Oct 30 14:11:58.953589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:58.952646    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:59.019632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:59.015715    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5365de31-dk558"]
Oct 30 14:11:59.231564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:59.218629    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:59.389745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:59.389697    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:59.391923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:59.390015    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:11:59.973626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:11:59.973561    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:00.231681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:00.231556    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:00.537463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:00.456698    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:00.537463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:00.531891    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:01.143286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.129071    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02bf1ede-db4e-4233-a082-ed75c386272c" path="/var/lib/kubelet/pods/02bf1ede-db4e-4233-a082-ed75c386272c/volumes"
Oct 30 14:12:01.154252 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:12:01.154215    4530 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.047s"
Oct 30 14:12:01.161589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.134386    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Oct 30 14:12:01.342582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.321726    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:01.691573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.690458    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:12:01.691573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.690510    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" event={"ID":"46b8b223-4999-4a21-a275-093515f6ccfe","Type":"ContainerStarted","Data":"c5c952300600f8cc0241c691999fa2d777915408746aaccbd24d7126422f6d1f"}
Oct 30 14:12:01.737715 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.737680    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:12:01.744967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.743173    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 14:12:01.744967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.744648    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"]
Oct 30 14:12:01.744967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.744694    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 14:12:01.768037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.768004    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"]
Oct 30 14:12:01.768519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.768494    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:12:01.768739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.768723    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:12:01.768912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.768890    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" event={"ID":"46b8b223-4999-4a21-a275-093515f6ccfe","Type":"ContainerStarted","Data":"be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89"}
Oct 30 14:12:01.769058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.769044    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:12:01.772468 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.772443    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:01.793354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.793290    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:01.793760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.793706    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:12:01.870514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.870463    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eaeaa01-bsrvh\" (UID: \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:01.870743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.870646    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eaeaa01-bsrvh\" (UID: \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:01.871126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.871094    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frz22\" (UniqueName: \"kubernetes.io/projected/d372b404-a53e-4865-88a3-b6cba5a3f5f8-kube-api-access-frz22\") pod \"maas-api-revoke-sub-5eaeaa01-bsrvh\" (UID: \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:01.875885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.875839    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" podStartSLOduration=6.875826081 podStartE2EDuration="6.875826081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:01.872141514 +0000 UTC m=+1386.904040454" watchObservedRunningTime="2026-10-30 14:12:01.875826081 +0000 UTC m=+1386.907725020"
Oct 30 14:12:01.974460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.974367    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eaeaa01-bsrvh\" (UID: \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:01.974659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.974467    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eaeaa01-bsrvh\" (UID: \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:01.974733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.974676    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frz22\" (UniqueName: \"kubernetes.io/projected/d372b404-a53e-4865-88a3-b6cba5a3f5f8-kube-api-access-frz22\") pod \"maas-api-revoke-sub-5eaeaa01-bsrvh\" (UID: \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:01.975865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.975830    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eaeaa01-bsrvh\" (UID: \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:01.977876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:01.977828    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eaeaa01-bsrvh\" (UID: \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:02.022643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:02.022607    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frz22\" (UniqueName: \"kubernetes.io/projected/d372b404-a53e-4865-88a3-b6cba5a3f5f8-kube-api-access-frz22\") pod \"maas-api-revoke-sub-5eaeaa01-bsrvh\" (UID: \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:02.107575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:02.105809    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:02.435355 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:02.435326    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd372b404_a53e_4865_88a3_b6cba5a3f5f8.slice/crio-bc7f3404f59a380d980b8164a994756aa12e32ba804d1d6c766036f01b219e8a WatchSource:0}: Error finding container bc7f3404f59a380d980b8164a994756aa12e32ba804d1d6c766036f01b219e8a: Status 404 returned error can't find the container with id bc7f3404f59a380d980b8164a994756aa12e32ba804d1d6c766036f01b219e8a
Oct 30 14:12:02.438248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:02.438223    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"]
Oct 30 14:12:02.742808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:02.742717    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 14:12:02.742808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:02.742767    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 14:12:02.744393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:02.744368    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh" event={"ID":"d372b404-a53e-4865-88a3-b6cba5a3f5f8","Type":"ContainerStarted","Data":"bc7f3404f59a380d980b8164a994756aa12e32ba804d1d6c766036f01b219e8a"}
Oct 30 14:12:03.742450 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:03.742414    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 14:12:03.742855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:03.742473    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 14:12:03.755059 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:03.755029    4530 generic.go:356] "Generic (PLEG): container finished" podID="d372b404-a53e-4865-88a3-b6cba5a3f5f8" containerID="0620aac666c48b1fbbc957c859fb6521a35db2bb3cdaa61c7b7e1116d59aa670" exitCode=0
Oct 30 14:12:03.755219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:03.755124    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh" event={"ID":"d372b404-a53e-4865-88a3-b6cba5a3f5f8","Type":"ContainerDied","Data":"0620aac666c48b1fbbc957c859fb6521a35db2bb3cdaa61c7b7e1116d59aa670"}
Oct 30 14:12:04.789377 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:04.789323    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 14:12:04.790048 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:04.790015    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 14:12:05.166087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.166064    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:05.176086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.176063    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-xks-ca\") pod \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\" (UID: \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\") "
Oct 30 14:12:05.176215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.176159    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d372b404-a53e-4865-88a3-b6cba5a3f5f8-kube-api-access-frz22\" (UniqueName: \"kubernetes.io/projected/d372b404-a53e-4865-88a3-b6cba5a3f5f8-kube-api-access-frz22\") pod \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\" (UID: \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\") "
Oct 30 14:12:05.176290 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.176221    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-openshift-ca\") pod \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\" (UID: \"d372b404-a53e-4865-88a3-b6cba5a3f5f8\") "
Oct 30 14:12:05.176667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.176628    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-openshift-ca" pod "d372b404-a53e-4865-88a3-b6cba5a3f5f8" (UID: "d372b404-a53e-4865-88a3-b6cba5a3f5f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:05.176939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.176920    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:05.178098 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.178075    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-xks-ca" pod "d372b404-a53e-4865-88a3-b6cba5a3f5f8" (UID: "d372b404-a53e-4865-88a3-b6cba5a3f5f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:05.178302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.178281    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d372b404-a53e-4865-88a3-b6cba5a3f5f8-kube-api-access-frz22" pod "d372b404-a53e-4865-88a3-b6cba5a3f5f8" (UID: "d372b404-a53e-4865-88a3-b6cba5a3f5f8"). InnerVolumeSpecName "kube-api-access-frz22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:05.278228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.278193    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d372b404-a53e-4865-88a3-b6cba5a3f5f8-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:05.278228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.278222    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frz22\" (UniqueName: \"kubernetes.io/projected/d372b404-a53e-4865-88a3-b6cba5a3f5f8-kube-api-access-frz22\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:05.742187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.742144    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 14:12:05.742383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.742204    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 14:12:05.912492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.912459    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"
Oct 30 14:12:05.913005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.912488    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh" event={"ID":"d372b404-a53e-4865-88a3-b6cba5a3f5f8","Type":"ContainerDied","Data":"bc7f3404f59a380d980b8164a994756aa12e32ba804d1d6c766036f01b219e8a"}
Oct 30 14:12:05.913005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:05.912583    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc7f3404f59a380d980b8164a994756aa12e32ba804d1d6c766036f01b219e8a"
Oct 30 14:12:06.755053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:06.755013    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 14:12:06.755361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:06.755325    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 14:12:07.054109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.054077    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"]
Oct 30 14:12:07.068079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.068042    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"]
Oct 30 14:12:07.077154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.077128    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22b72fe-jvwhl"]
Oct 30 14:12:07.106385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.106351    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a12cdd11-q4zd8"]
Oct 30 14:12:07.120703 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.120669    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8" path="/var/lib/kubelet/pods/8b2b24bc-d4f1-4ac2-bd28-0d0337a2ebb8/volumes"
Oct 30 14:12:07.121397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.121373    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4a2052e-e5b5-4a35-b33b-d094d2d9f800" path="/var/lib/kubelet/pods/e4a2052e-e5b5-4a35-b33b-d094d2d9f800/volumes"
Oct 30 14:12:07.164144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.164113    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"]
Oct 30 14:12:07.164864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.164848    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d372b404-a53e-4865-88a3-b6cba5a3f5f8" containerName="revoke-keys"
Oct 30 14:12:07.164919 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.164866    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d372b404-a53e-4865-88a3-b6cba5a3f5f8" containerName="revoke-keys"
Oct 30 14:12:07.165095 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.165074    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d372b404-a53e-4865-88a3-b6cba5a3f5f8" containerName="revoke-keys"
Oct 30 14:12:07.169896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.169878    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:07.192243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.190659    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:07.193645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.193615    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:12:07.203848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.200281    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"]
Oct 30 14:12:07.272097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.272061    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq6q4\" (UniqueName: \"kubernetes.io/projected/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-kube-api-access-xq6q4\") pod \"maas-api-revoke-sub-d06755c7-4zgjk\" (UID: \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:07.272264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.272236    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06755c7-4zgjk\" (UID: \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:07.272316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.272287    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06755c7-4zgjk\" (UID: \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:07.374419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.374318    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq6q4\" (UniqueName: \"kubernetes.io/projected/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-kube-api-access-xq6q4\") pod \"maas-api-revoke-sub-d06755c7-4zgjk\" (UID: \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:07.374419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.374417    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06755c7-4zgjk\" (UID: \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:07.374762 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.374459    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06755c7-4zgjk\" (UID: \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:07.375054 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.375032    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06755c7-4zgjk\" (UID: \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:07.376839 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.376814    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06755c7-4zgjk\" (UID: \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:07.401294 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.401262    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq6q4\" (UniqueName: \"kubernetes.io/projected/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-kube-api-access-xq6q4\") pod \"maas-api-revoke-sub-d06755c7-4zgjk\" (UID: \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:07.495329 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.495298    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:07.742775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.742667    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 14:12:07.742775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.742718    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 14:12:07.811946 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:07.811913    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbbedd4_c15c_4dd5_8197_2087e9a6979a.slice/crio-5aa0235fc856dd6475f39a1a0c925427aea63e5ef24cf5510bbd425b638a1d1b WatchSource:0}: Error finding container 5aa0235fc856dd6475f39a1a0c925427aea63e5ef24cf5510bbd425b638a1d1b: Status 404 returned error can't find the container with id 5aa0235fc856dd6475f39a1a0c925427aea63e5ef24cf5510bbd425b638a1d1b
Oct 30 14:12:07.815767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.815741    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"]
Oct 30 14:12:07.933195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:07.933162    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk" event={"ID":"8bbbedd4-c15c-4dd5-8197-2087e9a6979a","Type":"ContainerStarted","Data":"5aa0235fc856dd6475f39a1a0c925427aea63e5ef24cf5510bbd425b638a1d1b"}
Oct 30 14:12:08.747010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:08.746983    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:12:08.747481 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:08.747462    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:12:08.748282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:08.748258    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:12:08.945710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:08.945676    4530 generic.go:356] "Generic (PLEG): container finished" podID="8bbbedd4-c15c-4dd5-8197-2087e9a6979a" containerID="fe6e9085b1022d1ca06e87d3bdd23d9d92f6dfb9abf126a8618d05d0a8244f50" exitCode=0
Oct 30 14:12:08.945895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:08.945758    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk" event={"ID":"8bbbedd4-c15c-4dd5-8197-2087e9a6979a","Type":"ContainerDied","Data":"fe6e9085b1022d1ca06e87d3bdd23d9d92f6dfb9abf126a8618d05d0a8244f50"}
Oct 30 14:12:10.139038 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.139011    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:10.230951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.230864    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-xks-ca\") pod \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\" (UID: \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\") "
Oct 30 14:12:10.231103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.231000    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-openshift-ca\") pod \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\" (UID: \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\") "
Oct 30 14:12:10.231285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.231269    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-kube-api-access-xq6q4\" (UniqueName: \"kubernetes.io/projected/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-kube-api-access-xq6q4\") pod \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\" (UID: \"8bbbedd4-c15c-4dd5-8197-2087e9a6979a\") "
Oct 30 14:12:10.231355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.231316    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-openshift-ca" pod "8bbbedd4-c15c-4dd5-8197-2087e9a6979a" (UID: "8bbbedd4-c15c-4dd5-8197-2087e9a6979a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:10.231854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.231832    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:10.233109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.233081    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-xks-ca" pod "8bbbedd4-c15c-4dd5-8197-2087e9a6979a" (UID: "8bbbedd4-c15c-4dd5-8197-2087e9a6979a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:10.233503 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.233482    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-kube-api-access-xq6q4" pod "8bbbedd4-c15c-4dd5-8197-2087e9a6979a" (UID: "8bbbedd4-c15c-4dd5-8197-2087e9a6979a"). InnerVolumeSpecName "kube-api-access-xq6q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:10.333328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.333297    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq6q4\" (UniqueName: \"kubernetes.io/projected/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-kube-api-access-xq6q4\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:10.333328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.333325    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bbbedd4-c15c-4dd5-8197-2087e9a6979a-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:10.966552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.966512    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"
Oct 30 14:12:10.966715 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.966583    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk" event={"ID":"8bbbedd4-c15c-4dd5-8197-2087e9a6979a","Type":"ContainerDied","Data":"5aa0235fc856dd6475f39a1a0c925427aea63e5ef24cf5510bbd425b638a1d1b"}
Oct 30 14:12:10.966715 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:10.966653    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aa0235fc856dd6475f39a1a0c925427aea63e5ef24cf5510bbd425b638a1d1b"
Oct 30 14:12:11.369148 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:12:11.369113    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:12:12.815435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.815401    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"]
Oct 30 14:12:12.816651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.816425    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8bbbedd4-c15c-4dd5-8197-2087e9a6979a" containerName="revoke-keys"
Oct 30 14:12:12.816651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.816450    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bbbedd4-c15c-4dd5-8197-2087e9a6979a" containerName="revoke-keys"
Oct 30 14:12:12.817030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.817009    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bbbedd4-c15c-4dd5-8197-2087e9a6979a" containerName="revoke-keys"
Oct 30 14:12:12.821364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.821346    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:12.845530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.845498    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"]
Oct 30 14:12:12.849243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.849221    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-da21809f-gw-openshift-default-dockercfg-xk9c5\""
Oct 30 14:12:12.987405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.987372    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-socket\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:12.987580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.987427    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-certs\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:12.987580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.987471    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6lj2\" (UniqueName: \"kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-kube-api-access-n6lj2\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:12.987661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.987581    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-credential-socket\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:12.987661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.987613    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-data\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:12.987781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.987768    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-envoy\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:12.987822 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.987792    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istiod-ca-cert\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:12.987895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.987882    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-podinfo\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:12.987965 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:12.987954    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-token\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.078043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.077961    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"]
Oct 30 14:12:13.084133 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.084094    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.089840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.089809    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-token\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.089999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.089900    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-socket\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.089999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.089947    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-certs\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.089999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.089971    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6lj2\" (UniqueName: \"kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-kube-api-access-n6lj2\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.090172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.090005    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-credential-socket\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.090172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.090045    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-data\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.090280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.090243    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-envoy\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.090280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.090271    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istiod-ca-cert\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.090384 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.090375    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-podinfo\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.092229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.090852    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-data\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.092229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.091118    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-credential-socket\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.092229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.091188    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-socket\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.092229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.091520    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-certs\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.093763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.093674    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-podinfo\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.095246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.095224    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-envoy\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.116688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.108369    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ea3e7b1d-openshift-default-dockercfg-tgwfz\""
Oct 30 14:12:13.119489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.092960    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istiod-ca-cert\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.119489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.118574    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-token\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.119489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.118933    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6lj2\" (UniqueName: \"kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-kube-api-access-n6lj2\") pod \"e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") " pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.128915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.128890    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"]
Oct 30 14:12:13.150959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.150934    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:13.204562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.202676    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-socket\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.204562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.202794    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-certs\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.204562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.202922    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-credential-socket\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.204562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.202986    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnz9\" (UniqueName: \"kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-kube-api-access-jhnz9\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.204562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.203079    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-token\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.204562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.203142    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-data\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.204562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.203246    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-podinfo\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.204562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.203284    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istiod-ca-cert\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.204562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.203423    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-envoy\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.339129    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-token\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.342671    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-data\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.342799    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-podinfo\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.344508    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istiod-ca-cert\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.344683    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-envoy\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.344748    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-socket\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.344823    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-certs\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.344926    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-credential-socket\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.344984    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnz9\" (UniqueName: \"kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-kube-api-access-jhnz9\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.346248    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istiod-ca-cert\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.350889    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-data\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.352859    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-certs\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.353130    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-socket\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.358612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.353378    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-credential-socket\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.366852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.366792    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-envoy\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.396195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.396163    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-podinfo\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.396943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.396869    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-token\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.417560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.397746    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnz9\" (UniqueName: \"kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-kube-api-access-jhnz9\") pod \"e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") " pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:13.462252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:13.461042    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:14.001835 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:14.001792    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a77bcbc_6491_4e7f_9842_e4ea8968a1bf.slice/crio-8aac426dc443868f44acb20fa60788e0a046eb5f6ebb48b2797bf93f974671ae WatchSource:0}: Error finding container 8aac426dc443868f44acb20fa60788e0a046eb5f6ebb48b2797bf93f974671ae: Status 404 returned error can't find the container with id 8aac426dc443868f44acb20fa60788e0a046eb5f6ebb48b2797bf93f974671ae
Oct 30 14:12:14.004925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:14.004870    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:12:14.005022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:14.004979    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:12:14.005069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:14.005032    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:12:14.008373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:14.008235    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"]
Oct 30 14:12:14.040449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:14.040419    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"]
Oct 30 14:12:14.046026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:14.046004    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99469b37-vqs7r"]
Oct 30 14:12:14.203715 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:14.203687    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9979cc_ae59_4c43_abfe_88937c76fb2b.slice/crio-34d349268c7991e783e4e6e182fbbc3045192134dcbcdae75604c620813e7e18 WatchSource:0}: Error finding container 34d349268c7991e783e4e6e182fbbc3045192134dcbcdae75604c620813e7e18: Status 404 returned error can't find the container with id 34d349268c7991e783e4e6e182fbbc3045192134dcbcdae75604c620813e7e18
Oct 30 14:12:14.204674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:14.204631    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"]
Oct 30 14:12:14.206737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:14.206705    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:12:14.206833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:14.206768    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:12:14.206833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:14.206801    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:12:15.007607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:15.007575    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" event={"ID":"8e9979cc-ae59-4c43-abfe-88937c76fb2b","Type":"ContainerStarted","Data":"0412c56bf8fc0e24127c45de69f867feb24b8fb81760376a89bdaa56aae15f5c"}
Oct 30 14:12:15.008049 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:15.007634    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" event={"ID":"8e9979cc-ae59-4c43-abfe-88937c76fb2b","Type":"ContainerStarted","Data":"34d349268c7991e783e4e6e182fbbc3045192134dcbcdae75604c620813e7e18"}
Oct 30 14:12:15.009311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:15.009282    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" event={"ID":"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf","Type":"ContainerStarted","Data":"7b44104c4852e9462ffc11defe3c373e5d3f25ddb625ce02e0c0c9d00c485be0"}
Oct 30 14:12:15.009405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:15.009332    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" event={"ID":"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf","Type":"ContainerStarted","Data":"8aac426dc443868f44acb20fa60788e0a046eb5f6ebb48b2797bf93f974671ae"}
Oct 30 14:12:15.081143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:15.081082    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" podStartSLOduration=2.08106442 podStartE2EDuration="2.08106442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:15.07167604 +0000 UTC m=+1400.103574980" watchObservedRunningTime="2026-10-30 14:12:15.08106442 +0000 UTC m=+1400.112963360"
Oct 30 14:12:15.116982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:15.116951    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3a5e790-56ad-4ad5-b69d-a65c874f7903" path="/var/lib/kubelet/pods/e3a5e790-56ad-4ad5-b69d-a65c874f7903/volumes"
Oct 30 14:12:15.126120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:15.126074    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" podStartSLOduration=3.12606296 podStartE2EDuration="3.12606296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:15.123287127 +0000 UTC m=+1400.155186065" watchObservedRunningTime="2026-10-30 14:12:15.12606296 +0000 UTC m=+1400.157961899"
Oct 30 14:12:15.151962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:15.151932    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:15.157426 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:15.157301    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:15.472568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:15.464013    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:15.487414 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:15.484959    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:16.018100 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:16.018060    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:16.018530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:16.018135    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:16.018838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:16.018813    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:12:16.018937 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:16.018881    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:12:17.633381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.633351    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"]
Oct 30 14:12:17.639874 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.639854    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:17.659285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.659257    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"]
Oct 30 14:12:17.664845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.664821    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:17.665014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.664861    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:12:17.755303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.755272    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fbc72ea-5kswb\" (UID: \"780aae55-c801-48ca-af16-80b5b8739b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:17.755466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.755325    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fbc72ea-5kswb\" (UID: \"780aae55-c801-48ca-af16-80b5b8739b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:17.755532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.755510    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59bql\" (UniqueName: \"kubernetes.io/projected/780aae55-c801-48ca-af16-80b5b8739b67-kube-api-access-59bql\") pod \"maas-api-revoke-sub-8fbc72ea-5kswb\" (UID: \"780aae55-c801-48ca-af16-80b5b8739b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:17.856602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.856537    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59bql\" (UniqueName: \"kubernetes.io/projected/780aae55-c801-48ca-af16-80b5b8739b67-kube-api-access-59bql\") pod \"maas-api-revoke-sub-8fbc72ea-5kswb\" (UID: \"780aae55-c801-48ca-af16-80b5b8739b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:17.856773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.856752    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fbc72ea-5kswb\" (UID: \"780aae55-c801-48ca-af16-80b5b8739b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:17.856852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.856834    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fbc72ea-5kswb\" (UID: \"780aae55-c801-48ca-af16-80b5b8739b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:17.857359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.857340    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fbc72ea-5kswb\" (UID: \"780aae55-c801-48ca-af16-80b5b8739b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:17.859136 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.859117    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fbc72ea-5kswb\" (UID: \"780aae55-c801-48ca-af16-80b5b8739b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:17.883070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.883045    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59bql\" (UniqueName: \"kubernetes.io/projected/780aae55-c801-48ca-af16-80b5b8739b67-kube-api-access-59bql\") pod \"maas-api-revoke-sub-8fbc72ea-5kswb\" (UID: \"780aae55-c801-48ca-af16-80b5b8739b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:17.954595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:17.954486    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:18.262925 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:18.262895    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780aae55_c801_48ca_af16_80b5b8739b67.slice/crio-7a01ab812ea5ce544d9f649da8d60615045bc115d21a558ca020fd0bd36547bc WatchSource:0}: Error finding container 7a01ab812ea5ce544d9f649da8d60615045bc115d21a558ca020fd0bd36547bc: Status 404 returned error can't find the container with id 7a01ab812ea5ce544d9f649da8d60615045bc115d21a558ca020fd0bd36547bc
Oct 30 14:12:18.266740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.266716    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"]
Oct 30 14:12:18.310726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.310706    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"]
Oct 30 14:12:18.315203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.315187    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"
Oct 30 14:12:18.338604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.335784    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"]
Oct 30 14:12:18.338604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.336795    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-da21809f\""
Oct 30 14:12:18.338604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.336849    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-da21809f-dockercfg-bl5ll\""
Oct 30 14:12:18.368839 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.368748    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb9017b-82b2-4962-8a4f-c70e795b19bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq\" (UID: \"1cb9017b-82b2-4962-8a4f-c70e795b19bc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"
Oct 30 14:12:18.369350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.369178    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdc8c\" (UniqueName: \"kubernetes.io/projected/1cb9017b-82b2-4962-8a4f-c70e795b19bc-kube-api-access-kdc8c\") pod \"payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq\" (UID: \"1cb9017b-82b2-4962-8a4f-c70e795b19bc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"
Oct 30 14:12:18.507565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.483665    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb9017b-82b2-4962-8a4f-c70e795b19bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq\" (UID: \"1cb9017b-82b2-4962-8a4f-c70e795b19bc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"
Oct 30 14:12:18.507565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.483773    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdc8c\" (UniqueName: \"kubernetes.io/projected/1cb9017b-82b2-4962-8a4f-c70e795b19bc-kube-api-access-kdc8c\") pod \"payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq\" (UID: \"1cb9017b-82b2-4962-8a4f-c70e795b19bc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"
Oct 30 14:12:18.507565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.485385    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb9017b-82b2-4962-8a4f-c70e795b19bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq\" (UID: \"1cb9017b-82b2-4962-8a4f-c70e795b19bc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"
Oct 30 14:12:18.575566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.570928    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdc8c\" (UniqueName: \"kubernetes.io/projected/1cb9017b-82b2-4962-8a4f-c70e795b19bc-kube-api-access-kdc8c\") pod \"payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq\" (UID: \"1cb9017b-82b2-4962-8a4f-c70e795b19bc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"
Oct 30 14:12:18.634566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.632638    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"]
Oct 30 14:12:18.681039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.637377    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"
Oct 30 14:12:18.742562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.740257    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"]
Oct 30 14:12:18.742562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.740390    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"
Oct 30 14:12:18.837567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.834526    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7\" (UID: \"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7\") " pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"
Oct 30 14:12:18.837567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.834629    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lndb2\" (UniqueName: \"kubernetes.io/projected/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-kube-api-access-lndb2\") pod \"payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7\" (UID: \"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7\") " pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"
Oct 30 14:12:18.937104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.937071    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7\" (UID: \"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7\") " pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"
Oct 30 14:12:18.937236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.937157    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndb2\" (UniqueName: \"kubernetes.io/projected/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-kube-api-access-lndb2\") pod \"payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7\" (UID: \"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7\") " pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"
Oct 30 14:12:18.938610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:18.938052    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7\" (UID: \"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7\") " pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"
Oct 30 14:12:19.002824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:19.000129    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lndb2\" (UniqueName: \"kubernetes.io/projected/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-kube-api-access-lndb2\") pod \"payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7\" (UID: \"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7\") " pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"
Oct 30 14:12:19.135620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:19.135487    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb" event={"ID":"780aae55-c801-48ca-af16-80b5b8739b67","Type":"ContainerStarted","Data":"3df0cc180fcfd71b61859a3efa544d7514d57c71536b1dc4e4be4b317c0d30c5"}
Oct 30 14:12:19.135620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:19.135587    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb" event={"ID":"780aae55-c801-48ca-af16-80b5b8739b67","Type":"ContainerStarted","Data":"7a01ab812ea5ce544d9f649da8d60615045bc115d21a558ca020fd0bd36547bc"}
Oct 30 14:12:19.178701 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:19.178671    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb9017b_82b2_4962_8a4f_c70e795b19bc.slice/crio-2c0ee269264574988b96e603fbf6bde1d8f1eae897e5fa28499804af20c10008 WatchSource:0}: Error finding container 2c0ee269264574988b96e603fbf6bde1d8f1eae897e5fa28499804af20c10008: Status 404 returned error can't find the container with id 2c0ee269264574988b96e603fbf6bde1d8f1eae897e5fa28499804af20c10008
Oct 30 14:12:19.182605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:19.182469    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"]
Oct 30 14:12:19.191527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:19.191475    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb" podStartSLOduration=2.191457432 podStartE2EDuration="2.191457432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:19.180449989 +0000 UTC m=+1404.212348928" watchObservedRunningTime="2026-10-30 14:12:19.191457432 +0000 UTC m=+1404.223356371"
Oct 30 14:12:19.211365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:19.211341    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"
Oct 30 14:12:19.528065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:19.528039    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"]
Oct 30 14:12:19.539371 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:19.539347    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d28cec1_fa9e_4614_b6d1_6f9fac5d5fc7.slice/crio-03e2e7c37e54242e659aa0160e61faf9983fbf62478b7a1a9c2be493b41c517d WatchSource:0}: Error finding container 03e2e7c37e54242e659aa0160e61faf9983fbf62478b7a1a9c2be493b41c517d: Status 404 returned error can't find the container with id 03e2e7c37e54242e659aa0160e61faf9983fbf62478b7a1a9c2be493b41c517d
Oct 30 14:12:20.146979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.146875    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq" event={"ID":"1cb9017b-82b2-4962-8a4f-c70e795b19bc","Type":"ContainerStarted","Data":"454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c"}
Oct 30 14:12:20.146979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.146936    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq" event={"ID":"1cb9017b-82b2-4962-8a4f-c70e795b19bc","Type":"ContainerStarted","Data":"2c0ee269264574988b96e603fbf6bde1d8f1eae897e5fa28499804af20c10008"}
Oct 30 14:12:20.146979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.146964    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"
Oct 30 14:12:20.148950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.148921    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7" event={"ID":"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7","Type":"ContainerStarted","Data":"c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d"}
Oct 30 14:12:20.149097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.148981    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"
Oct 30 14:12:20.149097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.149004    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7" event={"ID":"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7","Type":"ContainerStarted","Data":"03e2e7c37e54242e659aa0160e61faf9983fbf62478b7a1a9c2be493b41c517d"}
Oct 30 14:12:20.150845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.150819    4530 generic.go:356] "Generic (PLEG): container finished" podID="780aae55-c801-48ca-af16-80b5b8739b67" containerID="3df0cc180fcfd71b61859a3efa544d7514d57c71536b1dc4e4be4b317c0d30c5" exitCode=0
Oct 30 14:12:20.150973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.150906    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb" event={"ID":"780aae55-c801-48ca-af16-80b5b8739b67","Type":"ContainerDied","Data":"3df0cc180fcfd71b61859a3efa544d7514d57c71536b1dc4e4be4b317c0d30c5"}
Oct 30 14:12:20.155046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.155024    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"]
Oct 30 14:12:20.158617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.158601    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"
Oct 30 14:12:20.177172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.177151    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ea3e7b1d\""
Oct 30 14:12:20.177852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.177829    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"]
Oct 30 14:12:20.194529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.194484    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq" podStartSLOduration=2.194471981 podStartE2EDuration="2.194471981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:20.192908267 +0000 UTC m=+1405.224807211" watchObservedRunningTime="2026-10-30 14:12:20.194471981 +0000 UTC m=+1405.226370920"
Oct 30 14:12:20.286720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.286656    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7" podStartSLOduration=2.286637331 podStartE2EDuration="2.286637331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:20.284166598 +0000 UTC m=+1405.316065538" watchObservedRunningTime="2026-10-30 14:12:20.286637331 +0000 UTC m=+1405.318536278"
Oct 30 14:12:20.313253 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.313217    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-plugins-config-volume\") pod \"payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2\" (UID: \"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30\") " pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"
Oct 30 14:12:20.313405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.313283    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz86r\" (UniqueName: \"kubernetes.io/projected/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-kube-api-access-zz86r\") pod \"payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2\" (UID: \"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30\") " pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"
Oct 30 14:12:20.440613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.437117    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz86r\" (UniqueName: \"kubernetes.io/projected/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-kube-api-access-zz86r\") pod \"payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2\" (UID: \"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30\") " pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"
Oct 30 14:12:20.440613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.437623    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-plugins-config-volume\") pod \"payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2\" (UID: \"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30\") " pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"
Oct 30 14:12:20.440613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.438663    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-plugins-config-volume\") pod \"payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2\" (UID: \"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30\") " pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"
Oct 30 14:12:20.503566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.494748    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz86r\" (UniqueName: \"kubernetes.io/projected/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-kube-api-access-zz86r\") pod \"payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2\" (UID: \"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30\") " pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"
Oct 30 14:12:20.778628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:20.777975    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"
Oct 30 14:12:21.196684 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:21.196656    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"]
Oct 30 14:12:21.197009 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:21.196749    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa59e71_0a8c_4e33_8076_7e6a72fd4f30.slice/crio-a641f1670713ef3613aa456179d04d705dc6e454a775f6d6840ca7d9e577e948 WatchSource:0}: Error finding container a641f1670713ef3613aa456179d04d705dc6e454a775f6d6840ca7d9e577e948: Status 404 returned error can't find the container with id a641f1670713ef3613aa456179d04d705dc6e454a775f6d6840ca7d9e577e948
Oct 30 14:12:21.339906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:21.339874    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:12:21.374194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:21.374170    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:21.468281 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:21.468199    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-openshift-ca\") pod \"780aae55-c801-48ca-af16-80b5b8739b67\" (UID: \"780aae55-c801-48ca-af16-80b5b8739b67\") "
Oct 30 14:12:21.468432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:21.468372    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-xks-ca\") pod \"780aae55-c801-48ca-af16-80b5b8739b67\" (UID: \"780aae55-c801-48ca-af16-80b5b8739b67\") "
Oct 30 14:12:21.468563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:21.468431    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780aae55-c801-48ca-af16-80b5b8739b67-kube-api-access-59bql\" (UniqueName: \"kubernetes.io/projected/780aae55-c801-48ca-af16-80b5b8739b67-kube-api-access-59bql\") pod \"780aae55-c801-48ca-af16-80b5b8739b67\" (UID: \"780aae55-c801-48ca-af16-80b5b8739b67\") "
Oct 30 14:12:21.468659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:21.468633    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-openshift-ca" pod "780aae55-c801-48ca-af16-80b5b8739b67" (UID: "780aae55-c801-48ca-af16-80b5b8739b67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:21.469182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:21.469158    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:21.470674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:21.470649    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780aae55-c801-48ca-af16-80b5b8739b67-kube-api-access-59bql" pod "780aae55-c801-48ca-af16-80b5b8739b67" (UID: "780aae55-c801-48ca-af16-80b5b8739b67"). InnerVolumeSpecName "kube-api-access-59bql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:21.470825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:21.470806    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-xks-ca" pod "780aae55-c801-48ca-af16-80b5b8739b67" (UID: "780aae55-c801-48ca-af16-80b5b8739b67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:21.570219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:21.570186    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59bql\" (UniqueName: \"kubernetes.io/projected/780aae55-c801-48ca-af16-80b5b8739b67-kube-api-access-59bql\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:21.570219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:21.570215    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/780aae55-c801-48ca-af16-80b5b8739b67-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:22.294622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:22.277687    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2" event={"ID":"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30","Type":"ContainerStarted","Data":"2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a"}
Oct 30 14:12:22.294622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:22.277760    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2" event={"ID":"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30","Type":"ContainerStarted","Data":"a641f1670713ef3613aa456179d04d705dc6e454a775f6d6840ca7d9e577e948"}
Oct 30 14:12:22.303586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:22.295475    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"
Oct 30 14:12:22.307564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:22.306819    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb" event={"ID":"780aae55-c801-48ca-af16-80b5b8739b67","Type":"ContainerDied","Data":"7a01ab812ea5ce544d9f649da8d60615045bc115d21a558ca020fd0bd36547bc"}
Oct 30 14:12:22.307564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:22.306880    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a01ab812ea5ce544d9f649da8d60615045bc115d21a558ca020fd0bd36547bc"
Oct 30 14:12:22.307564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:22.306951    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"
Oct 30 14:12:22.460577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:22.460014    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2" podStartSLOduration=2.4599901920000002 podStartE2EDuration="2.459990192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:22.384072814 +0000 UTC m=+1407.415971754" watchObservedRunningTime="2026-10-30 14:12:22.459990192 +0000 UTC m=+1407.491889131"
Oct 30 14:12:23.882568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:23.881633    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:24.037563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:24.032455    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"]
Oct 30 14:12:24.061567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:24.060771    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f774f453-dsl6k"]
Oct 30 14:12:24.088565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:24.079380    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:24.266307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:24.259632    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:24.901057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:24.900460    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:24.932601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:24.915009    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:12:25.031841 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:25.031801    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:25.291623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:25.262103    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:25.453273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:25.453017    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e" path="/var/lib/kubelet/pods/b9cd24bc-8ca3-4d0f-b7fd-5e3794a9619e/volumes"
Oct 30 14:12:25.454904 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:25.454883    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:12:25.562936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:25.562214    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:12:25.596842 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:25.596812    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:12:25.637421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:25.635465    4530 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:12:25.637421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:25.635559    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:27.259049 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.259005    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"]
Oct 30 14:12:27.259787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.259771    4530 cpu_manager.go:414] "containerMap: removing container" podUID="780aae55-c801-48ca-af16-80b5b8739b67" containerName="revoke-keys"
Oct 30 14:12:27.259831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.259788    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780aae55-c801-48ca-af16-80b5b8739b67" containerName="revoke-keys"
Oct 30 14:12:27.259890 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.259881    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="780aae55-c801-48ca-af16-80b5b8739b67" containerName="revoke-keys"
Oct 30 14:12:27.263514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.263495    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:27.284621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.284589    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"]
Oct 30 14:12:27.289321 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.289297    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:27.289437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.289377    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:12:27.318610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.318579    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02e8778e-v4rrf\" (UID: \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:27.318739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.318719    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lct2k\" (UniqueName: \"kubernetes.io/projected/2c8c779c-e58e-4b7c-b408-e7893410f4e7-kube-api-access-lct2k\") pod \"maas-api-revoke-sub-02e8778e-v4rrf\" (UID: \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:27.318901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.318877    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02e8778e-v4rrf\" (UID: \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:27.421022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.420991    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02e8778e-v4rrf\" (UID: \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:27.421177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.421078    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lct2k\" (UniqueName: \"kubernetes.io/projected/2c8c779c-e58e-4b7c-b408-e7893410f4e7-kube-api-access-lct2k\") pod \"maas-api-revoke-sub-02e8778e-v4rrf\" (UID: \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:27.421177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.421137    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02e8778e-v4rrf\" (UID: \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:27.421818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.421796    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02e8778e-v4rrf\" (UID: \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:27.423482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.423452    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02e8778e-v4rrf\" (UID: \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:27.449886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.449862    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lct2k\" (UniqueName: \"kubernetes.io/projected/2c8c779c-e58e-4b7c-b408-e7893410f4e7-kube-api-access-lct2k\") pod \"maas-api-revoke-sub-02e8778e-v4rrf\" (UID: \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:27.585803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:27.585757    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:28.022414 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:28.022386    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8c779c_e58e_4b7c_b408_e7893410f4e7.slice/crio-afe68ff5256b0a68d77ee4aef103f5999d6d281ce2724567cac6f2a9b1ddf1d5 WatchSource:0}: Error finding container afe68ff5256b0a68d77ee4aef103f5999d6d281ce2724567cac6f2a9b1ddf1d5: Status 404 returned error can't find the container with id afe68ff5256b0a68d77ee4aef103f5999d6d281ce2724567cac6f2a9b1ddf1d5
Oct 30 14:12:28.028297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:28.028266    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"]
Oct 30 14:12:28.483811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:28.483778    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf" event={"ID":"2c8c779c-e58e-4b7c-b408-e7893410f4e7","Type":"ContainerStarted","Data":"4ea30a449e45e08031e679d67679dbe84c32765bb320afb613fc5837446bf682"}
Oct 30 14:12:28.484253 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:28.483841    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf" event={"ID":"2c8c779c-e58e-4b7c-b408-e7893410f4e7","Type":"ContainerStarted","Data":"afe68ff5256b0a68d77ee4aef103f5999d6d281ce2724567cac6f2a9b1ddf1d5"}
Oct 30 14:12:28.531101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:28.531055    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf" podStartSLOduration=1.531042128 podStartE2EDuration="1.531042128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:28.528978044 +0000 UTC m=+1413.560876983" watchObservedRunningTime="2026-10-30 14:12:28.531042128 +0000 UTC m=+1413.562941067"
Oct 30 14:12:29.568288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:29.567627    4530 generic.go:356] "Generic (PLEG): container finished" podID="2c8c779c-e58e-4b7c-b408-e7893410f4e7" containerID="4ea30a449e45e08031e679d67679dbe84c32765bb320afb613fc5837446bf682" exitCode=0
Oct 30 14:12:29.568288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:29.567794    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf" event={"ID":"2c8c779c-e58e-4b7c-b408-e7893410f4e7","Type":"ContainerDied","Data":"4ea30a449e45e08031e679d67679dbe84c32765bb320afb613fc5837446bf682"}
Oct 30 14:12:30.747606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:30.747581    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:30.810084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:30.809893    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-openshift-ca\") pod \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\" (UID: \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\") "
Oct 30 14:12:30.810224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:30.810165    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-xks-ca\") pod \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\" (UID: \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\") "
Oct 30 14:12:30.810224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:30.810219    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8c779c-e58e-4b7c-b408-e7893410f4e7-kube-api-access-lct2k\" (UniqueName: \"kubernetes.io/projected/2c8c779c-e58e-4b7c-b408-e7893410f4e7-kube-api-access-lct2k\") pod \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\" (UID: \"2c8c779c-e58e-4b7c-b408-e7893410f4e7\") "
Oct 30 14:12:30.810353 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:30.810288    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-openshift-ca" pod "2c8c779c-e58e-4b7c-b408-e7893410f4e7" (UID: "2c8c779c-e58e-4b7c-b408-e7893410f4e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:30.810956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:30.810932    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:30.812849 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:30.812823    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-xks-ca" pod "2c8c779c-e58e-4b7c-b408-e7893410f4e7" (UID: "2c8c779c-e58e-4b7c-b408-e7893410f4e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:30.812988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:30.812971    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8c779c-e58e-4b7c-b408-e7893410f4e7-kube-api-access-lct2k" pod "2c8c779c-e58e-4b7c-b408-e7893410f4e7" (UID: "2c8c779c-e58e-4b7c-b408-e7893410f4e7"). InnerVolumeSpecName "kube-api-access-lct2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:30.912492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:30.912469    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8c779c-e58e-4b7c-b408-e7893410f4e7-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:30.912492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:30.912491    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lct2k\" (UniqueName: \"kubernetes.io/projected/2c8c779c-e58e-4b7c-b408-e7893410f4e7-kube-api-access-lct2k\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:31.175300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:31.175269    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"
Oct 30 14:12:31.175471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:31.175403    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"
Oct 30 14:12:31.675744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:31.675703    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf" event={"ID":"2c8c779c-e58e-4b7c-b408-e7893410f4e7","Type":"ContainerDied","Data":"afe68ff5256b0a68d77ee4aef103f5999d6d281ce2724567cac6f2a9b1ddf1d5"}
Oct 30 14:12:31.675986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:31.675969    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afe68ff5256b0a68d77ee4aef103f5999d6d281ce2724567cac6f2a9b1ddf1d5"
Oct 30 14:12:31.676169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:31.676156    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"
Oct 30 14:12:34.256084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.255819    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"
Oct 30 14:12:34.364577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.363470    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"]
Oct 30 14:12:34.367653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.364939    4530 cpu_manager.go:414] "containerMap: removing container" podUID="2c8c779c-e58e-4b7c-b408-e7893410f4e7" containerName="revoke-keys"
Oct 30 14:12:34.367653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.364967    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8c779c-e58e-4b7c-b408-e7893410f4e7" containerName="revoke-keys"
Oct 30 14:12:34.367653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.365180    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8c779c-e58e-4b7c-b408-e7893410f4e7" containerName="revoke-keys"
Oct 30 14:12:34.377634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.373920    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:34.401536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.401502    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:34.420378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.420348    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:12:34.431806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.431780    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"]
Oct 30 14:12:34.441534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.441503    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd44525d-g5zt4\" (UID: \"f388cf8f-159b-4884-9789-83e69fb9feb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:34.462132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.462085    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgx4\" (UniqueName: \"kubernetes.io/projected/f388cf8f-159b-4884-9789-83e69fb9feb3-kube-api-access-bmgx4\") pod \"maas-api-revoke-sub-dd44525d-g5zt4\" (UID: \"f388cf8f-159b-4884-9789-83e69fb9feb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:34.462451 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.462432    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd44525d-g5zt4\" (UID: \"f388cf8f-159b-4884-9789-83e69fb9feb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:34.577918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.572137    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd44525d-g5zt4\" (UID: \"f388cf8f-159b-4884-9789-83e69fb9feb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:34.577918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.572437    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgx4\" (UniqueName: \"kubernetes.io/projected/f388cf8f-159b-4884-9789-83e69fb9feb3-kube-api-access-bmgx4\") pod \"maas-api-revoke-sub-dd44525d-g5zt4\" (UID: \"f388cf8f-159b-4884-9789-83e69fb9feb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:34.577918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.572480    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd44525d-g5zt4\" (UID: \"f388cf8f-159b-4884-9789-83e69fb9feb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:34.577918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.573402    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd44525d-g5zt4\" (UID: \"f388cf8f-159b-4884-9789-83e69fb9feb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:34.583583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.583556    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd44525d-g5zt4\" (UID: \"f388cf8f-159b-4884-9789-83e69fb9feb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:34.605909 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.605878    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgx4\" (UniqueName: \"kubernetes.io/projected/f388cf8f-159b-4884-9789-83e69fb9feb3-kube-api-access-bmgx4\") pod \"maas-api-revoke-sub-dd44525d-g5zt4\" (UID: \"f388cf8f-159b-4884-9789-83e69fb9feb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:34.768616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:34.765666    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:35.138897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:35.138393    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"]
Oct 30 14:12:35.882241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:35.882208    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4" event={"ID":"f388cf8f-159b-4884-9789-83e69fb9feb3","Type":"ContainerStarted","Data":"1327422b92d18b6227c6dc79a41ad16289707209a807c97eacc12f0b826bb1fb"}
Oct 30 14:12:35.882758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:35.882275    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4" event={"ID":"f388cf8f-159b-4884-9789-83e69fb9feb3","Type":"ContainerStarted","Data":"fca73535d0cba543401d310441298b83df061c1c5e608f880c871cf76c75fda7"}
Oct 30 14:12:35.934839 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:35.934789    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4" podStartSLOduration=1.934775592 podStartE2EDuration="1.934775592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:35.933044015 +0000 UTC m=+1420.964942955" watchObservedRunningTime="2026-10-30 14:12:35.934775592 +0000 UTC m=+1420.966674556"
Oct 30 14:12:36.893304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:36.893266    4530 generic.go:356] "Generic (PLEG): container finished" podID="f388cf8f-159b-4884-9789-83e69fb9feb3" containerID="1327422b92d18b6227c6dc79a41ad16289707209a807c97eacc12f0b826bb1fb" exitCode=0
Oct 30 14:12:36.893304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:36.893297    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4" event={"ID":"f388cf8f-159b-4884-9789-83e69fb9feb3","Type":"ContainerDied","Data":"1327422b92d18b6227c6dc79a41ad16289707209a807c97eacc12f0b826bb1fb"}
Oct 30 14:12:38.101748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.101721    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:38.277152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.277078    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-xks-ca\") pod \"f388cf8f-159b-4884-9789-83e69fb9feb3\" (UID: \"f388cf8f-159b-4884-9789-83e69fb9feb3\") "
Oct 30 14:12:38.277327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.277312    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f388cf8f-159b-4884-9789-83e69fb9feb3-kube-api-access-bmgx4\" (UniqueName: \"kubernetes.io/projected/f388cf8f-159b-4884-9789-83e69fb9feb3-kube-api-access-bmgx4\") pod \"f388cf8f-159b-4884-9789-83e69fb9feb3\" (UID: \"f388cf8f-159b-4884-9789-83e69fb9feb3\") "
Oct 30 14:12:38.277390 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.277377    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-openshift-ca\") pod \"f388cf8f-159b-4884-9789-83e69fb9feb3\" (UID: \"f388cf8f-159b-4884-9789-83e69fb9feb3\") "
Oct 30 14:12:38.277743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.277717    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-openshift-ca" pod "f388cf8f-159b-4884-9789-83e69fb9feb3" (UID: "f388cf8f-159b-4884-9789-83e69fb9feb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:38.278294 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.278273    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:38.279173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.279147    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-xks-ca" pod "f388cf8f-159b-4884-9789-83e69fb9feb3" (UID: "f388cf8f-159b-4884-9789-83e69fb9feb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:38.279508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.279483    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f388cf8f-159b-4884-9789-83e69fb9feb3-kube-api-access-bmgx4" pod "f388cf8f-159b-4884-9789-83e69fb9feb3" (UID: "f388cf8f-159b-4884-9789-83e69fb9feb3"). InnerVolumeSpecName "kube-api-access-bmgx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:38.379328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.379302    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmgx4\" (UniqueName: \"kubernetes.io/projected/f388cf8f-159b-4884-9789-83e69fb9feb3-kube-api-access-bmgx4\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:38.379328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.379328    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f388cf8f-159b-4884-9789-83e69fb9feb3-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:38.936266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.936232    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4" event={"ID":"f388cf8f-159b-4884-9789-83e69fb9feb3","Type":"ContainerDied","Data":"fca73535d0cba543401d310441298b83df061c1c5e608f880c871cf76c75fda7"}
Oct 30 14:12:38.936266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.936256    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"
Oct 30 14:12:38.936480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:38.936285    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fca73535d0cba543401d310441298b83df061c1c5e608f880c871cf76c75fda7"
Oct 30 14:12:41.368533 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:12:41.368486    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:12:44.583394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.583360    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"]
Oct 30 14:12:44.584116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.584099    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f388cf8f-159b-4884-9789-83e69fb9feb3" containerName="revoke-keys"
Oct 30 14:12:44.584174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.584118    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f388cf8f-159b-4884-9789-83e69fb9feb3" containerName="revoke-keys"
Oct 30 14:12:44.584276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.584266    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f388cf8f-159b-4884-9789-83e69fb9feb3" containerName="revoke-keys"
Oct 30 14:12:44.610014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.609850    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:44.620832 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.620803    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"]
Oct 30 14:12:44.631593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.629998    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:44.643227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.643195    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:12:44.677297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.677265    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abfaa22a-sjpgm\" (UID: \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:44.677455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.677360    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmcb\" (UniqueName: \"kubernetes.io/projected/24f8a961-f83d-4ae8-848a-f0b97bee82b3-kube-api-access-zqmcb\") pod \"maas-api-revoke-sub-abfaa22a-sjpgm\" (UID: \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:44.677597 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.677568    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abfaa22a-sjpgm\" (UID: \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:44.779060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.779029    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abfaa22a-sjpgm\" (UID: \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:44.779248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.779170    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abfaa22a-sjpgm\" (UID: \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:44.779322 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.779253    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmcb\" (UniqueName: \"kubernetes.io/projected/24f8a961-f83d-4ae8-848a-f0b97bee82b3-kube-api-access-zqmcb\") pod \"maas-api-revoke-sub-abfaa22a-sjpgm\" (UID: \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:44.779782 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.779754    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abfaa22a-sjpgm\" (UID: \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:44.781477 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.781453    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abfaa22a-sjpgm\" (UID: \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:44.806483 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.806458    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmcb\" (UniqueName: \"kubernetes.io/projected/24f8a961-f83d-4ae8-848a-f0b97bee82b3-kube-api-access-zqmcb\") pod \"maas-api-revoke-sub-abfaa22a-sjpgm\" (UID: \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:44.944536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:44.944448    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:45.275533 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:45.275499    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f8a961_f83d_4ae8_848a_f0b97bee82b3.slice/crio-109bec56c7fbca9771d9e9b9e2f0edc036e67c3283597b7d073f968651b23320 WatchSource:0}: Error finding container 109bec56c7fbca9771d9e9b9e2f0edc036e67c3283597b7d073f968651b23320: Status 404 returned error can't find the container with id 109bec56c7fbca9771d9e9b9e2f0edc036e67c3283597b7d073f968651b23320
Oct 30 14:12:45.280796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:45.280769    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"]
Oct 30 14:12:46.024110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:46.024018    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm" event={"ID":"24f8a961-f83d-4ae8-848a-f0b97bee82b3","Type":"ContainerStarted","Data":"5d21aee4955b2a767aeed92b1978856d9de894a2edfbc713858b14c7c35003f7"}
Oct 30 14:12:46.024110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:46.024074    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm" event={"ID":"24f8a961-f83d-4ae8-848a-f0b97bee82b3","Type":"ContainerStarted","Data":"109bec56c7fbca9771d9e9b9e2f0edc036e67c3283597b7d073f968651b23320"}
Oct 30 14:12:46.072045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:46.071996    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm" podStartSLOduration=2.071980661 podStartE2EDuration="2.071980661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:46.068899023 +0000 UTC m=+1431.100797955" watchObservedRunningTime="2026-10-30 14:12:46.071980661 +0000 UTC m=+1431.103879617"
Oct 30 14:12:47.034735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:47.034695    4530 generic.go:356] "Generic (PLEG): container finished" podID="24f8a961-f83d-4ae8-848a-f0b97bee82b3" containerID="5d21aee4955b2a767aeed92b1978856d9de894a2edfbc713858b14c7c35003f7" exitCode=0
Oct 30 14:12:47.035158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:47.034789    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm" event={"ID":"24f8a961-f83d-4ae8-848a-f0b97bee82b3","Type":"ContainerDied","Data":"5d21aee4955b2a767aeed92b1978856d9de894a2edfbc713858b14c7c35003f7"}
Oct 30 14:12:48.208120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:48.208096    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:48.345425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:48.345393    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-openshift-ca\") pod \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\" (UID: \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\") "
Oct 30 14:12:48.345637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:48.345531    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-xks-ca\") pod \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\" (UID: \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\") "
Oct 30 14:12:48.345723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:48.345700    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f8a961-f83d-4ae8-848a-f0b97bee82b3-kube-api-access-zqmcb\" (UniqueName: \"kubernetes.io/projected/24f8a961-f83d-4ae8-848a-f0b97bee82b3-kube-api-access-zqmcb\") pod \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\" (UID: \"24f8a961-f83d-4ae8-848a-f0b97bee82b3\") "
Oct 30 14:12:48.345792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:48.345767    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-openshift-ca" pod "24f8a961-f83d-4ae8-848a-f0b97bee82b3" (UID: "24f8a961-f83d-4ae8-848a-f0b97bee82b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:48.346361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:48.346339    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:48.347724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:48.347698    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f8a961-f83d-4ae8-848a-f0b97bee82b3-kube-api-access-zqmcb" pod "24f8a961-f83d-4ae8-848a-f0b97bee82b3" (UID: "24f8a961-f83d-4ae8-848a-f0b97bee82b3"). InnerVolumeSpecName "kube-api-access-zqmcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:48.347724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:48.347713    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-xks-ca" pod "24f8a961-f83d-4ae8-848a-f0b97bee82b3" (UID: "24f8a961-f83d-4ae8-848a-f0b97bee82b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:48.448176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:48.448141    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f8a961-f83d-4ae8-848a-f0b97bee82b3-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:48.448176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:48.448173    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqmcb\" (UniqueName: \"kubernetes.io/projected/24f8a961-f83d-4ae8-848a-f0b97bee82b3-kube-api-access-zqmcb\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:49.063156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:49.063124    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"
Oct 30 14:12:49.063343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:49.063115    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm" event={"ID":"24f8a961-f83d-4ae8-848a-f0b97bee82b3","Type":"ContainerDied","Data":"109bec56c7fbca9771d9e9b9e2f0edc036e67c3283597b7d073f968651b23320"}
Oct 30 14:12:49.063343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:49.063311    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="109bec56c7fbca9771d9e9b9e2f0edc036e67c3283597b7d073f968651b23320"
Oct 30 14:12:49.432881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:49.432786    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"]
Oct 30 14:12:49.434444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:49.434396    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy" containerID="cri-o://31849c9b60d69f30b8fc64cdef65c7105bdb1235cc58a6d8b2f8f7e1323d8d66" gracePeriod=30
Oct 30 14:12:49.458512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:49.458048    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"]
Oct 30 14:12:49.458512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:49.458308    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" podUID="61375fe0-b572-459f-991d-169aa49b28ef" containerName="istio-proxy" containerID="cri-o://7ed38d1c95d5fb3d090045b15b155365cf4d5ca08588d3d64f9135516b5aa02f" gracePeriod=30
Oct 30 14:12:49.515530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:49.515207    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"]
Oct 30 14:12:49.515530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:49.515445    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" podUID="b39d4963-4125-44ec-aa19-020b8e287167" containerName="istio-proxy" containerID="cri-o://9b4d864a7f4b23e0cc628a8bcdb420d1a0be0454d30ecec53d2321139e92cc94" gracePeriod=30
Oct 30 14:12:49.527660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:49.527321    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"]
Oct 30 14:12:49.527819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:49.527770    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy" containerID="cri-o://ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5" gracePeriod=30
Oct 30 14:12:51.604227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:51.597621    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:51.984622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:51.983774    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Oct 30 14:12:52.079619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:52.078664    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:52.488566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:52.470440    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:52.557591 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:52.543712    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:52.594568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:52.589948    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:12:52.620565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:52.610902    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:52.620565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:52.611464    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:12:53.021565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:53.017991    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:53.021565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:53.018682    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:12:53.590566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:53.561661    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:53.738306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:53.594676    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:12:53.738925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:53.738887    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:53.742616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:53.617095    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:53.777767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:53.595236    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:12:53.837123 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:53.837082    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:12:53.837432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:53.837401    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:53.902623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:53.899951    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-94bz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:12:53.902623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:53.900019    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" podUID="b39d4963-4125-44ec-aa19-020b8e287167" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:12:54.096567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:54.091156    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:54.226990 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:54.226906    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:12:54.227706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:54.227676    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:12:54.782620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:54.762875    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 14:12:54.782620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:54.773687    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:55.044699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:55.044593    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:12:55.105627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:55.104162    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:12:55.105627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:55.104231    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:55.105627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:55.104309    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Oct 30 14:12:55.807379 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:55.788658    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:55.829783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:55.829663    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:56.087891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:56.059431    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:12:56.138198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:56.128758    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:56.503152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:56.503101    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:12:56.503560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:56.503506    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:56.895566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:56.891634    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:56.974334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:56.974289    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:57.016245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.016164    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"]
Oct 30 14:12:57.020490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.020453    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd" podUID="e6723427-e816-42ce-96f7-841d0d6538e3" containerName="istio-proxy" containerID="cri-o://1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51" gracePeriod=30
Oct 30 14:12:57.031306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.031247    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v" podUID="b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 14:12:57.036737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.036707    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"]
Oct 30 14:12:57.037157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.037125    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" podUID="cacf3a53-2882-438c-90ca-389a46799113" containerName="istio-proxy" containerID="cri-o://fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801" gracePeriod=30
Oct 30 14:12:57.058528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.058072    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"]
Oct 30 14:12:57.058528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.058316    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" podUID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" containerName="istio-proxy" containerID="cri-o://90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de" gracePeriod=30
Oct 30 14:12:57.089380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.084752    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:12:57.165652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.144745    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:12:57.558640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.558599    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"]
Oct 30 14:12:57.559137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.559118    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:12:57.560088 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.560056    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerName="istio-proxy" containerID="cri-o://a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5" gracePeriod=30
Oct 30 14:12:57.670921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.670883    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" event={"ID":"61375fe0-b572-459f-991d-169aa49b28ef","Type":"ContainerDied","Data":"7ed38d1c95d5fb3d090045b15b155365cf4d5ca08588d3d64f9135516b5aa02f"}
Oct 30 14:12:57.676789 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.670861    4530 generic.go:356] "Generic (PLEG): container finished" podID="61375fe0-b572-459f-991d-169aa49b28ef" containerID="7ed38d1c95d5fb3d090045b15b155365cf4d5ca08588d3d64f9135516b5aa02f" exitCode=0
Oct 30 14:12:57.677206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.677188    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:12:57.774658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.766662    4530 generic.go:356] "Generic (PLEG): container finished" podID="c0bfc568-8bae-43fe-adde-909edea76146" containerID="ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5" exitCode=0
Oct 30 14:12:57.774658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.767012    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" event={"ID":"c0bfc568-8bae-43fe-adde-909edea76146","Type":"ContainerDied","Data":"ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5"}
Oct 30 14:12:57.837116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.837083    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:12:57.845306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.845270    4530 generic.go:356] "Generic (PLEG): container finished" podID="b39d4963-4125-44ec-aa19-020b8e287167" containerID="9b4d864a7f4b23e0cc628a8bcdb420d1a0be0454d30ecec53d2321139e92cc94" exitCode=0
Oct 30 14:12:57.845657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.845630    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" event={"ID":"b39d4963-4125-44ec-aa19-020b8e287167","Type":"ContainerDied","Data":"9b4d864a7f4b23e0cc628a8bcdb420d1a0be0454d30ecec53d2321139e92cc94"}
Oct 30 14:12:57.845943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.845919    4530 scope.go:122] "RemoveContainer" containerID="fb0496343822c7dabae06ca9080c89d176c15a697b5459512fea8abd8fe8ea89"
Oct 30 14:12:57.903048 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.887116    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:12:57.935922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.909203    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 14:12:57.935922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.909259    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 14:12:57.974926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.974891    4530 generic.go:356] "Generic (PLEG): container finished" podID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerID="31849c9b60d69f30b8fc64cdef65c7105bdb1235cc58a6d8b2f8f7e1323d8d66" exitCode=0
Oct 30 14:12:57.976401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:57.975909    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" event={"ID":"256b8107-4ad4-46c2-88f2-26c2c15fe866","Type":"ContainerDied","Data":"31849c9b60d69f30b8fc64cdef65c7105bdb1235cc58a6d8b2f8f7e1323d8d66"}
Oct 30 14:12:58.075157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.075133    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:12:58.341695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.297364    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-kube-api-access-dmjsc\" (UniqueName: \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-kube-api-access-dmjsc\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.341695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.297502    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61375fe0-b572-459f-991d-169aa49b28ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61375fe0-b572-459f-991d-169aa49b28ef-istiod-ca-cert\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.341695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.297600    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-socket\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.341695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.297701    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-data\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.341695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.297746    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-certs\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.341695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.297811    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-envoy\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.341695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.297864    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61375fe0-b572-459f-991d-169aa49b28ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61375fe0-b572-459f-991d-169aa49b28ef-istio-podinfo\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.341695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.297931    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-credential-socket\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.341695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.298126    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-istio-token\" (UniqueName: \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-istio-token\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.393736 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.393690    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-data" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.402449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.402413    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-socket" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.403090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.403061    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61375fe0-b572-459f-991d-169aa49b28ef-istiod-ca-cert" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:58.405947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.405656    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-credential-socket" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.489198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.451224    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61375fe0-b572-459f-991d-169aa49b28ef-istio-podinfo" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:12:58.489198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.457924    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-istio-token" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:58.489198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.465495    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-kube-api-access-dmjsc" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "kube-api-access-dmjsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:58.489198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.474513    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-envoy" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.489198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.479513    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-certs" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.537412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.537359    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-kube-api-access-dmjsc\" (UniqueName: \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-kube-api-access-dmjsc\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.538054 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:58.537769    4530 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/61375fe0-b572-459f-991d-169aa49b28ef/volumes/kubernetes.io~projected/kube-api-access-dmjsc
Oct 30 14:12:58.538054 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.538019    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-kube-api-access-dmjsc" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "kube-api-access-dmjsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:58.551759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.551716    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61375fe0-b572-459f-991d-169aa49b28ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61375fe0-b572-459f-991d-169aa49b28ef-istiod-ca-cert\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.559585 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:58.552074    4530 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/61375fe0-b572-459f-991d-169aa49b28ef/volumes/kubernetes.io~configmap/istiod-ca-cert
Oct 30 14:12:58.559585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.554298    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61375fe0-b572-459f-991d-169aa49b28ef-istiod-ca-cert" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:58.559920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.559903    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-socket\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.560674    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-certs\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.560752    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-envoy\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.560806    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61375fe0-b572-459f-991d-169aa49b28ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61375fe0-b572-459f-991d-169aa49b28ef-istio-podinfo\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.560879    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-credential-socket\") pod \"61375fe0-b572-459f-991d-169aa49b28ef\" (UID: \"61375fe0-b572-459f-991d-169aa49b28ef\") "
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.561802    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61375fe0-b572-459f-991d-169aa49b28ef-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.561827    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.561842    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.561856    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmjsc\" (UniqueName: \"kubernetes.io/projected/61375fe0-b572-459f-991d-169aa49b28ef-kube-api-access-dmjsc\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:58.560072    4530 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/61375fe0-b572-459f-991d-169aa49b28ef/volumes/kubernetes.io~empty-dir/workload-socket
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.561933    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-socket" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:58.561919    4530 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/61375fe0-b572-459f-991d-169aa49b28ef/volumes/kubernetes.io~empty-dir/credential-socket
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.561957    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-credential-socket" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:58.562015    4530 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/61375fe0-b572-459f-991d-169aa49b28ef/volumes/kubernetes.io~empty-dir/workload-certs
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.562026    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-certs" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:58.562065    4530 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/61375fe0-b572-459f-991d-169aa49b28ef/volumes/kubernetes.io~empty-dir/istio-envoy
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.562073    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-envoy" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:58.562127    4530 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/61375fe0-b572-459f-991d-169aa49b28ef/volumes/kubernetes.io~downward-api/istio-podinfo
Oct 30 14:12:58.569278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.562134    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61375fe0-b572-459f-991d-169aa49b28ef-istio-podinfo" pod "61375fe0-b572-459f-991d-169aa49b28ef" (UID: "61375fe0-b572-459f-991d-169aa49b28ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:12:58.635870 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.584747    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 14:12:58.653071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.653045    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:12:58.661908 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.661873    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"]
Oct 30 14:12:58.663002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.662977    4530 cpu_manager.go:414] "containerMap: removing container" podUID="61375fe0-b572-459f-991d-169aa49b28ef" containerName="istio-proxy"
Oct 30 14:12:58.663171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.663156    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61375fe0-b572-459f-991d-169aa49b28ef" containerName="istio-proxy"
Oct 30 14:12:58.663281 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.663271    4530 cpu_manager.go:414] "containerMap: removing container" podUID="24f8a961-f83d-4ae8-848a-f0b97bee82b3" containerName="revoke-keys"
Oct 30 14:12:58.663365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.663355    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f8a961-f83d-4ae8-848a-f0b97bee82b3" containerName="revoke-keys"
Oct 30 14:12:58.663493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.663466    4530 cpu_manager.go:414] "containerMap: removing container" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy"
Oct 30 14:12:58.663618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.663092    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-data\") pod \"b39d4963-4125-44ec-aa19-020b8e287167\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") "
Oct 30 14:12:58.663838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.663813    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39d4963-4125-44ec-aa19-020b8e287167-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39d4963-4125-44ec-aa19-020b8e287167-istiod-ca-cert\") pod \"b39d4963-4125-44ec-aa19-020b8e287167\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") "
Oct 30 14:12:58.663923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.663886    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-credential-socket\") pod \"b39d4963-4125-44ec-aa19-020b8e287167\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") "
Oct 30 14:12:58.663992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.663954    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b39d4963-4125-44ec-aa19-020b8e287167-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39d4963-4125-44ec-aa19-020b8e287167-istio-podinfo\") pod \"b39d4963-4125-44ec-aa19-020b8e287167\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") "
Oct 30 14:12:58.663992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.663986    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-envoy\") pod \"b39d4963-4125-44ec-aa19-020b8e287167\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") "
Oct 30 14:12:58.664143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.663330    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-data" pod "b39d4963-4125-44ec-aa19-020b8e287167" (UID: "b39d4963-4125-44ec-aa19-020b8e287167"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.664256 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.664163    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-kube-api-access-tqx5z\" (UniqueName: \"kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-kube-api-access-tqx5z\") pod \"b39d4963-4125-44ec-aa19-020b8e287167\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") "
Oct 30 14:12:58.664406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.664390    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-istio-token\" (UniqueName: \"kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-istio-token\") pod \"b39d4963-4125-44ec-aa19-020b8e287167\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") "
Oct 30 14:12:58.664519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.663606    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy"
Oct 30 14:12:58.665477 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.665456    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b39d4963-4125-44ec-aa19-020b8e287167" containerName="istio-proxy"
Oct 30 14:12:58.665736 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.665718    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39d4963-4125-44ec-aa19-020b8e287167" containerName="istio-proxy"
Oct 30 14:12:58.665805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.664775    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-socket" pod "b39d4963-4125-44ec-aa19-020b8e287167" (UID: "b39d4963-4125-44ec-aa19-020b8e287167"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.665805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.665179    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-credential-socket" pod "b39d4963-4125-44ec-aa19-020b8e287167" (UID: "b39d4963-4125-44ec-aa19-020b8e287167"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.665926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.665820    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b39d4963-4125-44ec-aa19-020b8e287167" containerName="istio-proxy"
Oct 30 14:12:58.665926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.665829    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39d4963-4125-44ec-aa19-020b8e287167" containerName="istio-proxy"
Oct 30 14:12:58.665926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.664559    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-socket\") pod \"b39d4963-4125-44ec-aa19-020b8e287167\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") "
Oct 30 14:12:58.666067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.665923    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-certs\") pod \"b39d4963-4125-44ec-aa19-020b8e287167\" (UID: \"b39d4963-4125-44ec-aa19-020b8e287167\") "
Oct 30 14:12:58.666722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.666704    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39d4963-4125-44ec-aa19-020b8e287167-istiod-ca-cert" pod "b39d4963-4125-44ec-aa19-020b8e287167" (UID: "b39d4963-4125-44ec-aa19-020b8e287167"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:58.667091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667064    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-certs" pod "b39d4963-4125-44ec-aa19-020b8e287167" (UID: "b39d4963-4125-44ec-aa19-020b8e287167"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.667158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667060    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39d4963-4125-44ec-aa19-020b8e287167" containerName="istio-proxy"
Oct 30 14:12:58.667158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667135    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39d4963-4125-44ec-aa19-020b8e287167" containerName="istio-proxy"
Oct 30 14:12:58.667227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667158    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="61375fe0-b572-459f-991d-169aa49b28ef" containerName="istio-proxy"
Oct 30 14:12:58.667227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667172    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f8a961-f83d-4ae8-848a-f0b97bee82b3" containerName="revoke-keys"
Oct 30 14:12:58.667227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667184    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" containerName="istio-proxy"
Oct 30 14:12:58.667404 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.666726    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.667493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667481    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.667564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667555    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.667626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667619    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.667677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667671    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.667723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667717    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61375fe0-b572-459f-991d-169aa49b28ef-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.667777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667769    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61375fe0-b572-459f-991d-169aa49b28ef-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.667830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.667823    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.669050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.668970    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-envoy" pod "b39d4963-4125-44ec-aa19-020b8e287167" (UID: "b39d4963-4125-44ec-aa19-020b8e287167"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.669312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.669289    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-kube-api-access-tqx5z" pod "b39d4963-4125-44ec-aa19-020b8e287167" (UID: "b39d4963-4125-44ec-aa19-020b8e287167"). InnerVolumeSpecName "kube-api-access-tqx5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:58.669766 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.669750    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-istio-token" pod "b39d4963-4125-44ec-aa19-020b8e287167" (UID: "b39d4963-4125-44ec-aa19-020b8e287167"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:58.671265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.671238    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b39d4963-4125-44ec-aa19-020b8e287167-istio-podinfo" pod "b39d4963-4125-44ec-aa19-020b8e287167" (UID: "b39d4963-4125-44ec-aa19-020b8e287167"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:12:58.672305 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.672286    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"]
Oct 30 14:12:58.672408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.672395    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:12:58.713884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.697344    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:58.713884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.697496    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:12:58.768440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.768409    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-token\" (UniqueName: \"kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-token\") pod \"256b8107-4ad4-46c2-88f2-26c2c15fe866\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") "
Oct 30 14:12:58.768604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.768449    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-kube-api-access-srfj7\" (UniqueName: \"kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-kube-api-access-srfj7\") pod \"256b8107-4ad4-46c2-88f2-26c2c15fe866\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") "
Oct 30 14:12:58.768604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.768482    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-socket\") pod \"256b8107-4ad4-46c2-88f2-26c2c15fe866\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") "
Oct 30 14:12:58.768604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.768502    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-certs\") pod \"256b8107-4ad4-46c2-88f2-26c2c15fe866\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") "
Oct 30 14:12:58.768604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.768530    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-podinfo\") pod \"256b8107-4ad4-46c2-88f2-26c2c15fe866\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") "
Oct 30 14:12:58.768880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.768628    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-credential-socket\") pod \"256b8107-4ad4-46c2-88f2-26c2c15fe866\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") "
Oct 30 14:12:58.768880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.768702    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-envoy\") pod \"256b8107-4ad4-46c2-88f2-26c2c15fe866\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") "
Oct 30 14:12:58.768880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.768809    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/256b8107-4ad4-46c2-88f2-26c2c15fe866-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256b8107-4ad4-46c2-88f2-26c2c15fe866-istiod-ca-cert\") pod \"256b8107-4ad4-46c2-88f2-26c2c15fe866\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") "
Oct 30 14:12:58.769045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.768887    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-data\") pod \"256b8107-4ad4-46c2-88f2-26c2c15fe866\" (UID: \"256b8107-4ad4-46c2-88f2-26c2c15fe866\") "
Oct 30 14:12:58.769376 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.769254    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-certs" pod "256b8107-4ad4-46c2-88f2-26c2c15fe866" (UID: "256b8107-4ad4-46c2-88f2-26c2c15fe866"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.769376 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.769284    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ded77db-tlqwl\" (UID: \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.769404    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nv6h\" (UniqueName: \"kubernetes.io/projected/ebf3233f-658d-458c-b6e3-a1ac8b215c95-kube-api-access-4nv6h\") pod \"maas-api-revoke-sub-9ded77db-tlqwl\" (UID: \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.769611    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-socket" pod "256b8107-4ad4-46c2-88f2-26c2c15fe866" (UID: "256b8107-4ad4-46c2-88f2-26c2c15fe866"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.769706    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256b8107-4ad4-46c2-88f2-26c2c15fe866-istiod-ca-cert" pod "256b8107-4ad4-46c2-88f2-26c2c15fe866" (UID: "256b8107-4ad4-46c2-88f2-26c2c15fe866"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.769853    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-data" pod "256b8107-4ad4-46c2-88f2-26c2c15fe866" (UID: "256b8107-4ad4-46c2-88f2-26c2c15fe866"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.769944    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ded77db-tlqwl\" (UID: \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.770253    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.770296    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.770315    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqx5z\" (UniqueName: \"kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-kube-api-access-tqx5z\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.770329    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39d4963-4125-44ec-aa19-020b8e287167-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.770344    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.770376    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256b8107-4ad4-46c2-88f2-26c2c15fe866-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.770391    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39d4963-4125-44ec-aa19-020b8e287167-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.770403    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.770417    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39d4963-4125-44ec-aa19-020b8e287167-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.770447    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39d4963-4125-44ec-aa19-020b8e287167-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.770490    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-credential-socket" pod "256b8107-4ad4-46c2-88f2-26c2c15fe866" (UID: "256b8107-4ad4-46c2-88f2-26c2c15fe866"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.773750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.773036    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-podinfo" pod "256b8107-4ad4-46c2-88f2-26c2c15fe866" (UID: "256b8107-4ad4-46c2-88f2-26c2c15fe866"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:12:58.774782 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.774192    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-envoy" pod "256b8107-4ad4-46c2-88f2-26c2c15fe866" (UID: "256b8107-4ad4-46c2-88f2-26c2c15fe866"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.810578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.792742    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:12:58.810578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.803956    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-token" pod "256b8107-4ad4-46c2-88f2-26c2c15fe866" (UID: "256b8107-4ad4-46c2-88f2-26c2c15fe866"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:58.867170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.867118    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-kube-api-access-srfj7" pod "256b8107-4ad4-46c2-88f2-26c2c15fe866" (UID: "256b8107-4ad4-46c2-88f2-26c2c15fe866"). InnerVolumeSpecName "kube-api-access-srfj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:58.871497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.871465    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-credential-socket\") pod \"c0bfc568-8bae-43fe-adde-909edea76146\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") "
Oct 30 14:12:58.871727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.871534    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-certs\") pod \"c0bfc568-8bae-43fe-adde-909edea76146\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") "
Oct 30 14:12:58.871727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.871669    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-kube-api-access-l7t47\" (UniqueName: \"kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-kube-api-access-l7t47\") pod \"c0bfc568-8bae-43fe-adde-909edea76146\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") "
Oct 30 14:12:58.871727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.871724    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0bfc568-8bae-43fe-adde-909edea76146-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bfc568-8bae-43fe-adde-909edea76146-istiod-ca-cert\") pod \"c0bfc568-8bae-43fe-adde-909edea76146\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") "
Oct 30 14:12:58.871953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.871780    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0bfc568-8bae-43fe-adde-909edea76146-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bfc568-8bae-43fe-adde-909edea76146-istio-podinfo\") pod \"c0bfc568-8bae-43fe-adde-909edea76146\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") "
Oct 30 14:12:58.871953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.871871    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-data\") pod \"c0bfc568-8bae-43fe-adde-909edea76146\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") "
Oct 30 14:12:58.871953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.871886    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-credential-socket" pod "c0bfc568-8bae-43fe-adde-909edea76146" (UID: "c0bfc568-8bae-43fe-adde-909edea76146"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.871953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.871923    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-istio-token\" (UniqueName: \"kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-istio-token\") pod \"c0bfc568-8bae-43fe-adde-909edea76146\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") "
Oct 30 14:12:58.872158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.871954    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-certs" pod "c0bfc568-8bae-43fe-adde-909edea76146" (UID: "c0bfc568-8bae-43fe-adde-909edea76146"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.872158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.872048    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-envoy\") pod \"c0bfc568-8bae-43fe-adde-909edea76146\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") "
Oct 30 14:12:58.872158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.872096    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-socket\") pod \"c0bfc568-8bae-43fe-adde-909edea76146\" (UID: \"c0bfc568-8bae-43fe-adde-909edea76146\") "
Oct 30 14:12:58.872301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.872261    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bfc568-8bae-43fe-adde-909edea76146-istiod-ca-cert" pod "c0bfc568-8bae-43fe-adde-909edea76146" (UID: "c0bfc568-8bae-43fe-adde-909edea76146"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:58.872496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.872473    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ded77db-tlqwl\" (UID: \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:12:58.872711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.872688    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nv6h\" (UniqueName: \"kubernetes.io/projected/ebf3233f-658d-458c-b6e3-a1ac8b215c95-kube-api-access-4nv6h\") pod \"maas-api-revoke-sub-9ded77db-tlqwl\" (UID: \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:12:58.873008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.872990    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ded77db-tlqwl\" (UID: \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:12:58.873186 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.873161    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-socket" pod "c0bfc568-8bae-43fe-adde-909edea76146" (UID: "c0bfc568-8bae-43fe-adde-909edea76146"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.873296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.873280    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.873374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.873302    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.873374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.873316    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.873374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.873332    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.873374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.873347    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srfj7\" (UniqueName: \"kubernetes.io/projected/256b8107-4ad4-46c2-88f2-26c2c15fe866-kube-api-access-srfj7\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.873374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.873360    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256b8107-4ad4-46c2-88f2-26c2c15fe866-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.873374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.873374    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bfc568-8bae-43fe-adde-909edea76146-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.873730 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.873388    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256b8107-4ad4-46c2-88f2-26c2c15fe866-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.874588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.874396    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ded77db-tlqwl\" (UID: \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:12:58.874795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.874772    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-data" pod "c0bfc568-8bae-43fe-adde-909edea76146" (UID: "c0bfc568-8bae-43fe-adde-909edea76146"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.875041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.875016    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-istio-token" pod "c0bfc568-8bae-43fe-adde-909edea76146" (UID: "c0bfc568-8bae-43fe-adde-909edea76146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:58.875976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.875940    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-envoy" pod "c0bfc568-8bae-43fe-adde-909edea76146" (UID: "c0bfc568-8bae-43fe-adde-909edea76146"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.878349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.878321    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-kube-api-access-l7t47" pod "c0bfc568-8bae-43fe-adde-909edea76146" (UID: "c0bfc568-8bae-43fe-adde-909edea76146"). InnerVolumeSpecName "kube-api-access-l7t47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:58.878757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.878697    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ded77db-tlqwl\" (UID: \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:12:58.936919 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.934852    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0bfc568-8bae-43fe-adde-909edea76146-istio-podinfo" pod "c0bfc568-8bae-43fe-adde-909edea76146" (UID: "c0bfc568-8bae-43fe-adde-909edea76146"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:12:58.938285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.938227    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nv6h\" (UniqueName: \"kubernetes.io/projected/ebf3233f-658d-458c-b6e3-a1ac8b215c95-kube-api-access-4nv6h\") pod \"maas-api-revoke-sub-9ded77db-tlqwl\" (UID: \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:12:58.957179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.957154    4530 scope.go:122] "RemoveContainer" containerID="9fe678cf72af0778416bd42b2770809e6402db1b4947243229a334bab94892a2"
Oct 30 14:12:58.969037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.968976    4530 scope.go:122] "RemoveContainer" containerID="8da7b5505d26d471cac4bad856e3dd0b35da769b99bc13b606dfc642b8edbe0c"
Oct 30 14:12:58.975168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.975094    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.975168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.975124    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.975168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.975138    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.975168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.975150    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bfc568-8bae-43fe-adde-909edea76146-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.975168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.975163    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7t47\" (UniqueName: \"kubernetes.io/projected/c0bfc568-8bae-43fe-adde-909edea76146-kube-api-access-l7t47\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.975389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.975176    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bfc568-8bae-43fe-adde-909edea76146-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.979109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:58.979081    4530 scope.go:122] "RemoveContainer" containerID="b2396cde600d5639280aac21873c254654a44d4ef280133168b557bca31bd06b"
Oct 30 14:12:59.060595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.060516    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:12:59.061701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.061670    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s" event={"ID":"61375fe0-b572-459f-991d-169aa49b28ef","Type":"ContainerDied","Data":"770098f2ad5f210dbbaab5164d9a2a960cedcbe17989e8b97a7b7776af614b62"}
Oct 30 14:12:59.061920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.061899    4530 scope.go:122] "RemoveContainer" containerID="7ed38d1c95d5fb3d090045b15b155365cf4d5ca08588d3d64f9135516b5aa02f"
Oct 30 14:12:59.062096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.061916    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"
Oct 30 14:12:59.066097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.066073    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"
Oct 30 14:12:59.066779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.066082    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm" event={"ID":"c0bfc568-8bae-43fe-adde-909edea76146","Type":"ContainerDied","Data":"84fa087db8ccb07835fcf5bb2b83753bbcbb16e2191f0cbb359f283cf90d9a11"}
Oct 30 14:12:59.071756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.070649    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7" event={"ID":"b39d4963-4125-44ec-aa19-020b8e287167","Type":"ContainerDied","Data":"ddb3ff09e06c1427c0644fa1a05e526f82e3f276ed5e3e45032e50bcf9815e6b"}
Oct 30 14:12:59.071756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.070755    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"
Oct 30 14:12:59.074106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.074087    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"
Oct 30 14:12:59.101600 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.074122    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l" event={"ID":"256b8107-4ad4-46c2-88f2-26c2c15fe866","Type":"ContainerDied","Data":"4ad05b5b75742ba96305c15f4be0d01e2d4369b0226c64b81c81890507dfd89d"}
Oct 30 14:12:59.140921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.139130    4530 scope.go:122] "RemoveContainer" containerID="6d206a67a2983c402aedbd17f5a006da1db1c8af57475aaddab0da8c8bae3040"
Oct 30 14:12:59.142282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.142262    4530 scope.go:122] "RemoveContainer" containerID="ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5"
Oct 30 14:12:59.164563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.164074    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"]
Oct 30 14:12:59.173455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.173427    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wstfm"]
Oct 30 14:12:59.178954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.178859    4530 scope.go:122] "RemoveContainer" containerID="9b4d864a7f4b23e0cc628a8bcdb420d1a0be0454d30ecec53d2321139e92cc94"
Oct 30 14:12:59.252885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.252819    4530 scope.go:122] "RemoveContainer" containerID="9b4d864a7f4b23e0cc628a8bcdb420d1a0be0454d30ecec53d2321139e92cc94"
Oct 30 14:12:59.266338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.266315    4530 scope.go:122] "RemoveContainer" containerID="ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5"
Oct 30 14:12:59.266667 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:12:59.266640    4530 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_maas-default-gateway-openshift-default-c9ffffc7f-94bz7_openshift-ingress_b39d4963-4125-44ec-aa19-020b8e287167_1 in pod sandbox ddb3ff09e06c1427c0644fa1a05e526f82e3f276ed5e3e45032e50bcf9815e6b from index: no such id: '9b4d864a7f4b23e0cc628a8bcdb420d1a0be0454d30ecec53d2321139e92cc94'" containerID="9b4d864a7f4b23e0cc628a8bcdb420d1a0be0454d30ecec53d2321139e92cc94"
Oct 30 14:12:59.266761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.266677    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4d864a7f4b23e0cc628a8bcdb420d1a0be0454d30ecec53d2321139e92cc94"} err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_maas-default-gateway-openshift-default-c9ffffc7f-94bz7_openshift-ingress_b39d4963-4125-44ec-aa19-020b8e287167_1 in pod sandbox ddb3ff09e06c1427c0644fa1a05e526f82e3f276ed5e3e45032e50bcf9815e6b from index: no such id: '9b4d864a7f4b23e0cc628a8bcdb420d1a0be0454d30ecec53d2321139e92cc94'"
Oct 30 14:12:59.266761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.266695    4530 scope.go:122] "RemoveContainer" containerID="31849c9b60d69f30b8fc64cdef65c7105bdb1235cc58a6d8b2f8f7e1323d8d66"
Oct 30 14:12:59.266761 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:12:59.266726    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5\": container with ID starting with ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5 not found: ID does not exist" containerID="ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5"
Oct 30 14:12:59.266761 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:12:59.266743    4530 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5\": rpc error: code = NotFound desc = could not find container \"ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5\": container with ID starting with ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5 not found: ID does not exist" containerID="ef41c8e95e3ffa964825d60edf0ae82311d14026f7db986f55a91d1377c50ab5"
Oct 30 14:12:59.349555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.349510    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"]
Oct 30 14:12:59.359683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.359648    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ww87s"]
Oct 30 14:12:59.371770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.371742    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"]
Oct 30 14:12:59.379449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.379424    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tfw9l"]
Oct 30 14:12:59.447657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.447632    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"]
Oct 30 14:12:59.453342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.453314    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-94bz7"]
Oct 30 14:12:59.568117 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:12:59.568089    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf3233f_658d_458c_b6e3_a1ac8b215c95.slice/crio-1db75eb8eb8da084090d4d7d80cb6398b0f8749b8fa47f3a57ebf218901e126c WatchSource:0}: Error finding container 1db75eb8eb8da084090d4d7d80cb6398b0f8749b8fa47f3a57ebf218901e126c: Status 404 returned error can't find the container with id 1db75eb8eb8da084090d4d7d80cb6398b0f8749b8fa47f3a57ebf218901e126c
Oct 30 14:12:59.574161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:12:59.574134    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"]
Oct 30 14:13:00.291213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:00.291173    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl" event={"ID":"ebf3233f-658d-458c-b6e3-a1ac8b215c95","Type":"ContainerStarted","Data":"1db75eb8eb8da084090d4d7d80cb6398b0f8749b8fa47f3a57ebf218901e126c"}
Oct 30 14:13:00.413565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:00.403899    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl" podStartSLOduration=2.403873813 podStartE2EDuration="2.403873813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:00.347960469 +0000 UTC m=+1445.379859409" watchObservedRunningTime="2026-10-30 14:13:00.403873813 +0000 UTC m=+1445.435772755"
Oct 30 14:13:01.140816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.140785    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256b8107-4ad4-46c2-88f2-26c2c15fe866" path="/var/lib/kubelet/pods/256b8107-4ad4-46c2-88f2-26c2c15fe866/volumes"
Oct 30 14:13:01.141605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.141586    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61375fe0-b572-459f-991d-169aa49b28ef" path="/var/lib/kubelet/pods/61375fe0-b572-459f-991d-169aa49b28ef/volumes"
Oct 30 14:13:01.142146 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.142133    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39d4963-4125-44ec-aa19-020b8e287167" path="/var/lib/kubelet/pods/b39d4963-4125-44ec-aa19-020b8e287167/volumes"
Oct 30 14:13:01.143058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.143042    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0bfc568-8bae-43fe-adde-909edea76146" path="/var/lib/kubelet/pods/c0bfc568-8bae-43fe-adde-909edea76146/volumes"
Oct 30 14:13:01.314835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.314794    4530 generic.go:356] "Generic (PLEG): container finished" podID="ebf3233f-658d-458c-b6e3-a1ac8b215c95" containerID="6f4be218a3b043515fa90c2e4543428bfb8410e8e04fadeccd943166492fff92" exitCode=0
Oct 30 14:13:01.315208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.314884    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl" event={"ID":"ebf3233f-658d-458c-b6e3-a1ac8b215c95","Type":"ContainerDied","Data":"6f4be218a3b043515fa90c2e4543428bfb8410e8e04fadeccd943166492fff92"}
Oct 30 14:13:01.358894 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.358864    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:13:01.359042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.358921    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd" podUID="e6723427-e816-42ce-96f7-841d0d6538e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:13:01.572101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.572022    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:13:01.572101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.572082    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" podUID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:13:01.740684 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.740638    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:13:01.740880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.740712    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" podUID="cacf3a53-2882-438c-90ca-389a46799113" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:13:01.786056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.786015    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:13:01.786236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:01.786080    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:13:02.676476 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:02.676441    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:13:02.852930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:02.848441    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf3233f-658d-458c-b6e3-a1ac8b215c95-kube-api-access-4nv6h\" (UniqueName: \"kubernetes.io/projected/ebf3233f-658d-458c-b6e3-a1ac8b215c95-kube-api-access-4nv6h\") pod \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\" (UID: \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\") "
Oct 30 14:13:02.852930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:02.848571    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-openshift-ca\") pod \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\" (UID: \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\") "
Oct 30 14:13:02.852930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:02.848661    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-xks-ca\") pod \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\" (UID: \"ebf3233f-658d-458c-b6e3-a1ac8b215c95\") "
Oct 30 14:13:02.853912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:02.853855    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-openshift-ca" pod "ebf3233f-658d-458c-b6e3-a1ac8b215c95" (UID: "ebf3233f-658d-458c-b6e3-a1ac8b215c95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:02.886462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:02.886175    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf3233f-658d-458c-b6e3-a1ac8b215c95-kube-api-access-4nv6h" pod "ebf3233f-658d-458c-b6e3-a1ac8b215c95" (UID: "ebf3233f-658d-458c-b6e3-a1ac8b215c95"). InnerVolumeSpecName "kube-api-access-4nv6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:02.886462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:02.886264    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-xks-ca" pod "ebf3233f-658d-458c-b6e3-a1ac8b215c95" (UID: "ebf3233f-658d-458c-b6e3-a1ac8b215c95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:02.962369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:02.955784    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nv6h\" (UniqueName: \"kubernetes.io/projected/ebf3233f-658d-458c-b6e3-a1ac8b215c95-kube-api-access-4nv6h\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:02.962369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:02.955816    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:02.962369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:02.955832    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf3233f-658d-458c-b6e3-a1ac8b215c95-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:02.962369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:02.958760    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:13:03.029264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.029235    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:13:03.058718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.056972    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-istio-token\" (UniqueName: \"kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-istio-token\") pod \"cacf3a53-2882-438c-90ca-389a46799113\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") "
Oct 30 14:13:03.058718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.057101    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-credential-socket\") pod \"cacf3a53-2882-438c-90ca-389a46799113\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") "
Oct 30 14:13:03.058718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.057153    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-envoy\") pod \"cacf3a53-2882-438c-90ca-389a46799113\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") "
Oct 30 14:13:03.058718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.057289    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-kube-api-access-rhbsd\" (UniqueName: \"kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-kube-api-access-rhbsd\") pod \"cacf3a53-2882-438c-90ca-389a46799113\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") "
Oct 30 14:13:03.058718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.057322    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cacf3a53-2882-438c-90ca-389a46799113-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf3a53-2882-438c-90ca-389a46799113-istio-podinfo\") pod \"cacf3a53-2882-438c-90ca-389a46799113\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") "
Oct 30 14:13:03.058718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.057359    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-certs\") pod \"cacf3a53-2882-438c-90ca-389a46799113\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") "
Oct 30 14:13:03.058718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.057478    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cacf3a53-2882-438c-90ca-389a46799113-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf3a53-2882-438c-90ca-389a46799113-istiod-ca-cert\") pod \"cacf3a53-2882-438c-90ca-389a46799113\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") "
Oct 30 14:13:03.058718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.057515    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-data\") pod \"cacf3a53-2882-438c-90ca-389a46799113\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") "
Oct 30 14:13:03.058718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.057684    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-socket\") pod \"cacf3a53-2882-438c-90ca-389a46799113\" (UID: \"cacf3a53-2882-438c-90ca-389a46799113\") "
Oct 30 14:13:03.059533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.059355    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-socket" pod "cacf3a53-2882-438c-90ca-389a46799113" (UID: "cacf3a53-2882-438c-90ca-389a46799113"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.060804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.060380    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-credential-socket" pod "cacf3a53-2882-438c-90ca-389a46799113" (UID: "cacf3a53-2882-438c-90ca-389a46799113"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.060804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.060767    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cacf3a53-2882-438c-90ca-389a46799113-istiod-ca-cert" pod "cacf3a53-2882-438c-90ca-389a46799113" (UID: "cacf3a53-2882-438c-90ca-389a46799113"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:03.061075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.061032    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-certs" pod "cacf3a53-2882-438c-90ca-389a46799113" (UID: "cacf3a53-2882-438c-90ca-389a46799113"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.061144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.061099    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-data" pod "cacf3a53-2882-438c-90ca-389a46799113" (UID: "cacf3a53-2882-438c-90ca-389a46799113"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.065352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.065318    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cacf3a53-2882-438c-90ca-389a46799113-istio-podinfo" pod "cacf3a53-2882-438c-90ca-389a46799113" (UID: "cacf3a53-2882-438c-90ca-389a46799113"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:13:03.065781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.065742    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-kube-api-access-rhbsd" pod "cacf3a53-2882-438c-90ca-389a46799113" (UID: "cacf3a53-2882-438c-90ca-389a46799113"). InnerVolumeSpecName "kube-api-access-rhbsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:03.066235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.066213    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-istio-token" pod "cacf3a53-2882-438c-90ca-389a46799113" (UID: "cacf3a53-2882-438c-90ca-389a46799113"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:03.067802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.067773    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-envoy" pod "cacf3a53-2882-438c-90ca-389a46799113" (UID: "cacf3a53-2882-438c-90ca-389a46799113"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.101391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.100309    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"]
Oct 30 14:13:03.105286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.105212    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cacf3a53-2882-438c-90ca-389a46799113" containerName="istio-proxy"
Oct 30 14:13:03.105286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.105244    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacf3a53-2882-438c-90ca-389a46799113" containerName="istio-proxy"
Oct 30 14:13:03.105286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.105266    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy"
Oct 30 14:13:03.105286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.105275    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy"
Oct 30 14:13:03.105286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.105290    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerName="istio-proxy"
Oct 30 14:13:03.105655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.105297    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerName="istio-proxy"
Oct 30 14:13:03.105655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.105323    4530 cpu_manager.go:414] "containerMap: removing container" podUID="ebf3233f-658d-458c-b6e3-a1ac8b215c95" containerName="revoke-keys"
Oct 30 14:13:03.105655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.105330    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf3233f-658d-458c-b6e3-a1ac8b215c95" containerName="revoke-keys"
Oct 30 14:13:03.105655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.105601    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cacf3a53-2882-438c-90ca-389a46799113" containerName="istio-proxy"
Oct 30 14:13:03.105655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.105620    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0bfc568-8bae-43fe-adde-909edea76146" containerName="istio-proxy"
Oct 30 14:13:03.105655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.105633    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerName="istio-proxy"
Oct 30 14:13:03.105655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.105646    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf3233f-658d-458c-b6e3-a1ac8b215c95" containerName="revoke-keys"
Oct 30 14:13:03.121721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.121692    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:03.159491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.159454    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-certs\") pod \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") "
Oct 30 14:13:03.159666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.159584    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-data\") pod \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") "
Oct 30 14:13:03.159666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.159628    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-podinfo\") pod \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") "
Oct 30 14:13:03.159798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.159669    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-credential-socket\") pod \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") "
Oct 30 14:13:03.159854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.159808    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-kube-api-access-pgq9j\" (UniqueName: \"kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-kube-api-access-pgq9j\") pod \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") "
Oct 30 14:13:03.159920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.159851    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istiod-ca-cert\") pod \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") "
Oct 30 14:13:03.159920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.159896    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-token\" (UniqueName: \"kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-token\") pod \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") "
Oct 30 14:13:03.160032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.159969    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-envoy\") pod \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") "
Oct 30 14:13:03.160107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.160086    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-socket\") pod \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\" (UID: \"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf\") "
Oct 30 14:13:03.160947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.160921    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhbsd\" (UniqueName: \"kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-kube-api-access-rhbsd\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.161057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.160951    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf3a53-2882-438c-90ca-389a46799113-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.161057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.160966    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.161057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.160980    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf3a53-2882-438c-90ca-389a46799113-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.161057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.160996    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.161057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.161008    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.161057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.161021    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacf3a53-2882-438c-90ca-389a46799113-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.161057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.161035    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.161057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.161049    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf3a53-2882-438c-90ca-389a46799113-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.161718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.161690    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istiod-ca-cert" pod "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" (UID: "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:03.162441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.162381    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-data" pod "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" (UID: "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.162441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.162400    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-certs" pod "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" (UID: "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.162626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.162462    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-credential-socket" pod "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" (UID: "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.162626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.162581    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-socket" pod "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" (UID: "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.171559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.168529    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"]
Oct 30 14:13:03.182273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.181912    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:13:03.186865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.186764    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-envoy" pod "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" (UID: "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.186865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.186798    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-kube-api-access-pgq9j" pod "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" (UID: "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf"). InnerVolumeSpecName "kube-api-access-pgq9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:03.187912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.187143    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-token" pod "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" (UID: "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:03.189984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.189956    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-podinfo" pod "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" (UID: "9a8d1f93-81d0-4f01-b25f-939c53b4b6cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:13:03.236663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.236618    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:13:03.264934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.263826    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpskt\" (UniqueName: \"kubernetes.io/projected/795be2b8-287e-4fc5-b101-0c10cc5dcdad-kube-api-access-qpskt\") pod \"maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j\" (UID: \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:03.264934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.263936    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j\" (UID: \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:03.264934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.264123    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j\" (UID: \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:03.264934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.264337    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.264934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.264357    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.264934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.264372    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.264934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.264385    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.264934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.264400    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgq9j\" (UniqueName: \"kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-kube-api-access-pgq9j\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.264934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.264414    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.264934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.264427    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.264934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.264440    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.264934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.264453    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.343415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.343372    4530 generic.go:356] "Generic (PLEG): container finished" podID="cacf3a53-2882-438c-90ca-389a46799113" containerID="fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801" exitCode=0
Oct 30 14:13:03.343626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.343468    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" event={"ID":"cacf3a53-2882-438c-90ca-389a46799113","Type":"ContainerDied","Data":"fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801"}
Oct 30 14:13:03.343626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.343523    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc" event={"ID":"cacf3a53-2882-438c-90ca-389a46799113","Type":"ContainerDied","Data":"97c434334757df75ee633a49cb1c37da963c8905dfcc65065da539b079954f1f"}
Oct 30 14:13:03.343626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.343576    4530 scope.go:122] "RemoveContainer" containerID="fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801"
Oct 30 14:13:03.343804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.343758    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"
Oct 30 14:13:03.352359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.350796    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl" event={"ID":"ebf3233f-658d-458c-b6e3-a1ac8b215c95","Type":"ContainerDied","Data":"1db75eb8eb8da084090d4d7d80cb6398b0f8749b8fa47f3a57ebf218901e126c"}
Oct 30 14:13:03.352359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.350852    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db75eb8eb8da084090d4d7d80cb6398b0f8749b8fa47f3a57ebf218901e126c"
Oct 30 14:13:03.352359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.350910    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"
Oct 30 14:13:03.355346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.355094    4530 generic.go:356] "Generic (PLEG): container finished" podID="e6723427-e816-42ce-96f7-841d0d6538e3" containerID="1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51" exitCode=0
Oct 30 14:13:03.355346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.355149    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"
Oct 30 14:13:03.355346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.355227    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd" event={"ID":"e6723427-e816-42ce-96f7-841d0d6538e3","Type":"ContainerDied","Data":"1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51"}
Oct 30 14:13:03.355346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.355268    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd" event={"ID":"e6723427-e816-42ce-96f7-841d0d6538e3","Type":"ContainerDied","Data":"0310c8992f58635d2b2e0cb4f869774642003c850f5386e214363d1cb42933bc"}
Oct 30 14:13:03.362432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.357179    4530 generic.go:356] "Generic (PLEG): container finished" podID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" containerID="a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5" exitCode=0
Oct 30 14:13:03.362432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.357320    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" event={"ID":"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf","Type":"ContainerDied","Data":"a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5"}
Oct 30 14:13:03.362432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.357363    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b" event={"ID":"9a8d1f93-81d0-4f01-b25f-939c53b4b6cf","Type":"ContainerDied","Data":"1c0e4a0ea792f02991bbd9966768c57745ca3a65d9ec125d1d87aafb5ce824f1"}
Oct 30 14:13:03.362432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.357448    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"
Oct 30 14:13:03.365059 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.364935    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-podinfo\") pod \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") "
Oct 30 14:13:03.365176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.365104    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-credential-socket\") pod \"e6723427-e816-42ce-96f7-841d0d6538e3\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") "
Oct 30 14:13:03.365176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.365141    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-certs\") pod \"e6723427-e816-42ce-96f7-841d0d6538e3\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") "
Oct 30 14:13:03.365274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.365207    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-data\") pod \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") "
Oct 30 14:13:03.365944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.365607    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-credential-socket" pod "e6723427-e816-42ce-96f7-841d0d6538e3" (UID: "e6723427-e816-42ce-96f7-841d0d6538e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.365944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.365721    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-certs" pod "e6723427-e816-42ce-96f7-841d0d6538e3" (UID: "e6723427-e816-42ce-96f7-841d0d6538e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.365944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.365852    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-kube-api-access-skh2w\" (UniqueName: \"kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-kube-api-access-skh2w\") pod \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") "
Oct 30 14:13:03.365944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.365891    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-kube-api-access-mxv58\" (UniqueName: \"kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-kube-api-access-mxv58\") pod \"e6723427-e816-42ce-96f7-841d0d6538e3\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") "
Oct 30 14:13:03.365944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.365935    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-certs\") pod \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") "
Oct 30 14:13:03.366236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.365980    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-credential-socket\") pod \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") "
Oct 30 14:13:03.366236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.366031    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6723427-e816-42ce-96f7-841d0d6538e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6723427-e816-42ce-96f7-841d0d6538e3-istio-podinfo\") pod \"e6723427-e816-42ce-96f7-841d0d6538e3\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") "
Oct 30 14:13:03.366236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.366067    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6723427-e816-42ce-96f7-841d0d6538e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6723427-e816-42ce-96f7-841d0d6538e3-istiod-ca-cert\") pod \"e6723427-e816-42ce-96f7-841d0d6538e3\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") "
Oct 30 14:13:03.366236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.366106    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-data\") pod \"e6723427-e816-42ce-96f7-841d0d6538e3\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") "
Oct 30 14:13:03.366236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.366211    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-socket\") pod \"e6723427-e816-42ce-96f7-841d0d6538e3\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") "
Oct 30 14:13:03.366471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.366245    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-envoy\") pod \"e6723427-e816-42ce-96f7-841d0d6538e3\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") "
Oct 30 14:13:03.366471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.366322    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-token\" (UniqueName: \"kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-token\") pod \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") "
Oct 30 14:13:03.366471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.366365    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-envoy\") pod \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") "
Oct 30 14:13:03.366471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.366437    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istiod-ca-cert\") pod \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") "
Oct 30 14:13:03.366680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.366520    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-istio-token\" (UniqueName: \"kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-istio-token\") pod \"e6723427-e816-42ce-96f7-841d0d6538e3\" (UID: \"e6723427-e816-42ce-96f7-841d0d6538e3\") "
Oct 30 14:13:03.366680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.366609    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-socket\") pod \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\" (UID: \"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b\") "
Oct 30 14:13:03.366932 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.366908    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j\" (UID: \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:03.367160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.367142    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j\" (UID: \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:03.367461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.367440    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpskt\" (UniqueName: \"kubernetes.io/projected/795be2b8-287e-4fc5-b101-0c10cc5dcdad-kube-api-access-qpskt\") pod \"maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j\" (UID: \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:03.369860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.368440    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-data" pod "e6723427-e816-42ce-96f7-841d0d6538e3" (UID: "e6723427-e816-42ce-96f7-841d0d6538e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.369860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.368803    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6723427-e816-42ce-96f7-841d0d6538e3-istiod-ca-cert" pod "e6723427-e816-42ce-96f7-841d0d6538e3" (UID: "e6723427-e816-42ce-96f7-841d0d6538e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:03.369860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.369068    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-socket" pod "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" (UID: "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.369860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.369097    4530 generic.go:356] "Generic (PLEG): container finished" podID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" containerID="90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de" exitCode=0
Oct 30 14:13:03.369860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.369166    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" event={"ID":"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b","Type":"ContainerDied","Data":"90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de"}
Oct 30 14:13:03.369860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.369212    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn" event={"ID":"f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b","Type":"ContainerDied","Data":"1ac181415b5ad756b52687659bfca77ae57c8bfd65412e0bfcd9936d4c1cd482"}
Oct 30 14:13:03.369860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.369287    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"
Oct 30 14:13:03.369860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.369597    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-certs" pod "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" (UID: "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.370348 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.369899    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j\" (UID: \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:03.375005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.373489    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-kube-api-access-mxv58" pod "e6723427-e816-42ce-96f7-841d0d6538e3" (UID: "e6723427-e816-42ce-96f7-841d0d6538e3"). InnerVolumeSpecName "kube-api-access-mxv58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:03.375005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.373608    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-istio-token" pod "e6723427-e816-42ce-96f7-841d0d6538e3" (UID: "e6723427-e816-42ce-96f7-841d0d6538e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:03.375005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.373861    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6723427-e816-42ce-96f7-841d0d6538e3-istio-podinfo" pod "e6723427-e816-42ce-96f7-841d0d6538e3" (UID: "e6723427-e816-42ce-96f7-841d0d6538e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:13:03.375005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.374135    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-credential-socket" pod "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" (UID: "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.375005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.374313    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-data" pod "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" (UID: "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.375005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.374452    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-socket" pod "e6723427-e816-42ce-96f7-841d0d6538e3" (UID: "e6723427-e816-42ce-96f7-841d0d6538e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.375005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.374924    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istiod-ca-cert" pod "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" (UID: "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:03.376768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.375838    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j\" (UID: \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:03.376768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.375864    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-token" pod "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" (UID: "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:03.376768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.375927    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.376768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.375957    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.377423 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.377386    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-podinfo" pod "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" (UID: "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:13:03.378334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.378304    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-envoy" pod "e6723427-e816-42ce-96f7-841d0d6538e3" (UID: "e6723427-e816-42ce-96f7-841d0d6538e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.378869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.378834    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-kube-api-access-skh2w" pod "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" (UID: "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b"). InnerVolumeSpecName "kube-api-access-skh2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:03.380204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.380169    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-envoy" pod "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" (UID: "f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:03.389447 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.389419    4530 scope.go:122] "RemoveContainer" containerID="fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801"
Oct 30 14:13:03.389827 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:13:03.389802    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801\": container with ID starting with fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801 not found: ID does not exist" containerID="fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801"
Oct 30 14:13:03.389934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.389836    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801"} err="failed to get container status \"fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801\": rpc error: code = NotFound desc = could not find container \"fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801\": container with ID starting with fac219c7680a8b04a855b4180313bfc086315ee9186fa82d8f4aaa521c0e6801 not found: ID does not exist"
Oct 30 14:13:03.389934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.389854    4530 scope.go:122] "RemoveContainer" containerID="1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51"
Oct 30 14:13:03.423000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.420468    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpskt\" (UniqueName: \"kubernetes.io/projected/795be2b8-287e-4fc5-b101-0c10cc5dcdad-kube-api-access-qpskt\") pod \"maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j\" (UID: \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:03.432757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.432731    4530 scope.go:122] "RemoveContainer" containerID="1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51"
Oct 30 14:13:03.433216 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:13:03.433184    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51\": container with ID starting with 1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51 not found: ID does not exist" containerID="1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51"
Oct 30 14:13:03.433288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.433231    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51"} err="failed to get container status \"1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51\": rpc error: code = NotFound desc = could not find container \"1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51\": container with ID starting with 1ff22495b1a6469cba571329cdf3349deb997e61bee427f4268a4a2e50c98e51 not found: ID does not exist"
Oct 30 14:13:03.433288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.433258    4530 scope.go:122] "RemoveContainer" containerID="a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5"
Oct 30 14:13:03.445366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.445264    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:03.447411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.447385    4530 scope.go:122] "RemoveContainer" containerID="a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5"
Oct 30 14:13:03.448165 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:13:03.447814    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5\": container with ID starting with a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5 not found: ID does not exist" containerID="a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5"
Oct 30 14:13:03.448165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.447856    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5"} err="failed to get container status \"a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5\": rpc error: code = NotFound desc = could not find container \"a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5\": container with ID starting with a83e39ebda361d9882ef38bf5bb9ef25552b1657c424caaf6e78d47b21de5fa5 not found: ID does not exist"
Oct 30 14:13:03.448165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.447881    4530 scope.go:122] "RemoveContainer" containerID="90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de"
Oct 30 14:13:03.469691 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.469668    4530 scope.go:122] "RemoveContainer" containerID="90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de"
Oct 30 14:13:03.474233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.474193    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"]
Oct 30 14:13:03.474556 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:13:03.474512    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de\": container with ID starting with 90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de not found: ID does not exist" containerID="90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de"
Oct 30 14:13:03.474728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.474701    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de"} err="failed to get container status \"90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de\": rpc error: code = NotFound desc = could not find container \"90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de\": container with ID starting with 90fd4bac71f15e50a7ce20a465be5634a8fe3552ab1907151dbf35152f3257de not found: ID does not exist"
Oct 30 14:13:03.477706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477686    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6723427-e816-42ce-96f7-841d0d6538e3-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.477803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477712    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.477803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477723    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.477803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477732    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6723427-e816-42ce-96f7-841d0d6538e3-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.477803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477742    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.477803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477772    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.477803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477780    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.477803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477788    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.477803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477797    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.478189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477808    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.478189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477818    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.478189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477828    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skh2w\" (UniqueName: \"kubernetes.io/projected/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-kube-api-access-skh2w\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.478189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477838    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxv58\" (UniqueName: \"kubernetes.io/projected/e6723427-e816-42ce-96f7-841d0d6538e3-kube-api-access-mxv58\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.478189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477850    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.478189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477859    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.478189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.477867    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6723427-e816-42ce-96f7-841d0d6538e3-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:03.484105 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.484081    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4n9lc"]
Oct 30 14:13:03.540183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.539032    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"]
Oct 30 14:13:03.553792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.548478    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x676b"]
Oct 30 14:13:03.713850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.713801    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"]
Oct 30 14:13:03.719250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.719220    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-4gzzd"]
Oct 30 14:13:03.765129 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.765094    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"]
Oct 30 14:13:03.772826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.772795    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-ft9tn"]
Oct 30 14:13:03.790789 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:03.790757    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"]
Oct 30 14:13:03.811102 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:13:03.794586    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795be2b8_287e_4fc5_b101_0c10cc5dcdad.slice/crio-004c5b9fc7d72e725206a3888372f0aedeaa27cd91e9aa7ec4905c1b6ebfa205 WatchSource:0}: Error finding container 004c5b9fc7d72e725206a3888372f0aedeaa27cd91e9aa7ec4905c1b6ebfa205: Status 404 returned error can't find the container with id 004c5b9fc7d72e725206a3888372f0aedeaa27cd91e9aa7ec4905c1b6ebfa205
Oct 30 14:13:04.389090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:04.388988    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j" event={"ID":"795be2b8-287e-4fc5-b101-0c10cc5dcdad","Type":"ContainerStarted","Data":"b5edf4dcb7f1e1dc41fbcd63bd6060abc7ee010609699d284f14b3b10a7e5eb7"}
Oct 30 14:13:04.389090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:04.389061    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j" event={"ID":"795be2b8-287e-4fc5-b101-0c10cc5dcdad","Type":"ContainerStarted","Data":"004c5b9fc7d72e725206a3888372f0aedeaa27cd91e9aa7ec4905c1b6ebfa205"}
Oct 30 14:13:04.441104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:04.441030    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j" podStartSLOduration=1.441010886 podStartE2EDuration="1.441010886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:04.434414195 +0000 UTC m=+1449.466313137" watchObservedRunningTime="2026-10-30 14:13:04.441010886 +0000 UTC m=+1449.472909825"
Oct 30 14:13:05.117942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:05.117905    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8d1f93-81d0-4f01-b25f-939c53b4b6cf" path="/var/lib/kubelet/pods/9a8d1f93-81d0-4f01-b25f-939c53b4b6cf/volumes"
Oct 30 14:13:05.118651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:05.118631    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cacf3a53-2882-438c-90ca-389a46799113" path="/var/lib/kubelet/pods/cacf3a53-2882-438c-90ca-389a46799113/volumes"
Oct 30 14:13:05.119166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:05.119146    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6723427-e816-42ce-96f7-841d0d6538e3" path="/var/lib/kubelet/pods/e6723427-e816-42ce-96f7-841d0d6538e3/volumes"
Oct 30 14:13:05.119977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:05.119955    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" path="/var/lib/kubelet/pods/f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b/volumes"
Oct 30 14:13:05.408236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:05.408210    4530 generic.go:356] "Generic (PLEG): container finished" podID="795be2b8-287e-4fc5-b101-0c10cc5dcdad" containerID="b5edf4dcb7f1e1dc41fbcd63bd6060abc7ee010609699d284f14b3b10a7e5eb7" exitCode=0
Oct 30 14:13:05.408395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:05.408309    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j" event={"ID":"795be2b8-287e-4fc5-b101-0c10cc5dcdad","Type":"ContainerDied","Data":"b5edf4dcb7f1e1dc41fbcd63bd6060abc7ee010609699d284f14b3b10a7e5eb7"}
Oct 30 14:13:05.410403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:05.410379    4530 generic.go:356] "Generic (PLEG): container finished" podID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerID="7b44104c4852e9462ffc11defe3c373e5d3f25ddb625ce02e0c0c9d00c485be0" exitCode=0
Oct 30 14:13:05.410508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:05.410454    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" event={"ID":"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf","Type":"ContainerDied","Data":"7b44104c4852e9462ffc11defe3c373e5d3f25ddb625ce02e0c0c9d00c485be0"}
Oct 30 14:13:05.410843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:05.410829    4530 scope.go:122] "RemoveContainer" containerID="7b44104c4852e9462ffc11defe3c373e5d3f25ddb625ce02e0c0c9d00c485be0"
Oct 30 14:13:05.412160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:05.412133    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:05.412214 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:05.412201    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:05.412249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:05.412232    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:06.422084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:06.422036    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" event={"ID":"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf","Type":"ContainerStarted","Data":"d9dad1866d2f6b0ee2758ae88cd7f686c33ebd7d600a11b0396558f6e45ac59a"}
Oct 30 14:13:06.643854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:06.643830    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:06.744845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:06.744759    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-xks-ca\") pod \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\" (UID: \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\") "
Oct 30 14:13:06.745005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:06.744907    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795be2b8-287e-4fc5-b101-0c10cc5dcdad-kube-api-access-qpskt\" (UniqueName: \"kubernetes.io/projected/795be2b8-287e-4fc5-b101-0c10cc5dcdad-kube-api-access-qpskt\") pod \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\" (UID: \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\") "
Oct 30 14:13:06.745150 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:06.745132    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-openshift-ca\") pod \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\" (UID: \"795be2b8-287e-4fc5-b101-0c10cc5dcdad\") "
Oct 30 14:13:06.745600 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:06.745536    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-openshift-ca" pod "795be2b8-287e-4fc5-b101-0c10cc5dcdad" (UID: "795be2b8-287e-4fc5-b101-0c10cc5dcdad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:06.745984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:06.745958    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:06.747767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:06.747735    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-xks-ca" pod "795be2b8-287e-4fc5-b101-0c10cc5dcdad" (UID: "795be2b8-287e-4fc5-b101-0c10cc5dcdad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:06.747767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:06.747739    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795be2b8-287e-4fc5-b101-0c10cc5dcdad-kube-api-access-qpskt" pod "795be2b8-287e-4fc5-b101-0c10cc5dcdad" (UID: "795be2b8-287e-4fc5-b101-0c10cc5dcdad"). InnerVolumeSpecName "kube-api-access-qpskt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:06.847560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:06.847511    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpskt\" (UniqueName: \"kubernetes.io/projected/795be2b8-287e-4fc5-b101-0c10cc5dcdad-kube-api-access-qpskt\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:06.847560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:06.847568    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795be2b8-287e-4fc5-b101-0c10cc5dcdad-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:07.151937 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:07.151909    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:13:07.152268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:07.152027    4530 patch_prober.go:29] interesting pod/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 14:13:07.152268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:07.152073    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 14:13:07.433288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:07.433212    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"
Oct 30 14:13:07.433288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:07.433253    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j" event={"ID":"795be2b8-287e-4fc5-b101-0c10cc5dcdad","Type":"ContainerDied","Data":"004c5b9fc7d72e725206a3888372f0aedeaa27cd91e9aa7ec4905c1b6ebfa205"}
Oct 30 14:13:07.433849 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:07.433303    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="004c5b9fc7d72e725206a3888372f0aedeaa27cd91e9aa7ec4905c1b6ebfa205"
Oct 30 14:13:07.435238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:07.435208    4530 generic.go:356] "Generic (PLEG): container finished" podID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerID="0412c56bf8fc0e24127c45de69f867feb24b8fb81760376a89bdaa56aae15f5c" exitCode=0
Oct 30 14:13:07.435393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:07.435274    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" event={"ID":"8e9979cc-ae59-4c43-abfe-88937c76fb2b","Type":"ContainerDied","Data":"0412c56bf8fc0e24127c45de69f867feb24b8fb81760376a89bdaa56aae15f5c"}
Oct 30 14:13:07.436087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:07.436064    4530 scope.go:122] "RemoveContainer" containerID="0412c56bf8fc0e24127c45de69f867feb24b8fb81760376a89bdaa56aae15f5c"
Oct 30 14:13:07.437586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:07.437526    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:07.437661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:07.437648    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:07.437698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:07.437689    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:08.152003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.151968    4530 patch_prober.go:29] interesting pod/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 14:13:08.152184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.152065    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 14:13:08.446943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.446852    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" event={"ID":"8e9979cc-ae59-4c43-abfe-88937c76fb2b","Type":"ContainerStarted","Data":"50af43fa9ee96aa5787ac51ef5cc82b3d02cbbec5776ca1eda233af45848b6a3"}
Oct 30 14:13:08.463833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.463805    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:13:08.464028 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.464006    4530 patch_prober.go:29] interesting pod/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 14:13:08.464122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.464069    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 14:13:08.637282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.637241    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"]
Oct 30 14:13:08.637952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.637935    4530 cpu_manager.go:414] "containerMap: removing container" podUID="795be2b8-287e-4fc5-b101-0c10cc5dcdad" containerName="revoke-keys"
Oct 30 14:13:08.637952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.637951    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795be2b8-287e-4fc5-b101-0c10cc5dcdad" containerName="revoke-keys"
Oct 30 14:13:08.638115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.637969    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e6723427-e816-42ce-96f7-841d0d6538e3" containerName="istio-proxy"
Oct 30 14:13:08.638115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.637974    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6723427-e816-42ce-96f7-841d0d6538e3" containerName="istio-proxy"
Oct 30 14:13:08.638115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.638005    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" containerName="istio-proxy"
Oct 30 14:13:08.638115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.638010    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" containerName="istio-proxy"
Oct 30 14:13:08.638350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.638149    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="795be2b8-287e-4fc5-b101-0c10cc5dcdad" containerName="revoke-keys"
Oct 30 14:13:08.638350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.638162    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6723427-e816-42ce-96f7-841d0d6538e3" containerName="istio-proxy"
Oct 30 14:13:08.638350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.638173    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1650f8b-ce39-4004-84cf-c0f5dd8b0b5b" containerName="istio-proxy"
Oct 30 14:13:08.641273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.641254    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:08.661631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.660223    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"]
Oct 30 14:13:08.663676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.663643    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:13:08.663899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.663881    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:13:08.735628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.735502    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsg7v\" (UniqueName: \"kubernetes.io/projected/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-kube-api-access-wsg7v\") pod \"maas-api-revoke-sub-9760220e-hlqr4\" (UID: \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:08.735628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.735606    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9760220e-hlqr4\" (UID: \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:08.735880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.735642    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9760220e-hlqr4\" (UID: \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:08.839535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.838321    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsg7v\" (UniqueName: \"kubernetes.io/projected/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-kube-api-access-wsg7v\") pod \"maas-api-revoke-sub-9760220e-hlqr4\" (UID: \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:08.839535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.838437    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9760220e-hlqr4\" (UID: \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:08.839535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.838470    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9760220e-hlqr4\" (UID: \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:08.839814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.839694    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9760220e-hlqr4\" (UID: \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:08.841456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.841430    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9760220e-hlqr4\" (UID: \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:08.866247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.866217    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsg7v\" (UniqueName: \"kubernetes.io/projected/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-kube-api-access-wsg7v\") pod \"maas-api-revoke-sub-9760220e-hlqr4\" (UID: \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:08.957792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:08.957755    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:09.152929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:09.152893    4530 patch_prober.go:29] interesting pod/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 14:13:09.153120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:09.152956    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 14:13:09.335492 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:13:09.335454    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88ca0ca_4b55_4052_be9e_ed18a12a9d3a.slice/crio-fc0dd3ce23815f3f2411d0cb1c54a63946d45fb87b8e0d07db8b1c0839efb7bb WatchSource:0}: Error finding container fc0dd3ce23815f3f2411d0cb1c54a63946d45fb87b8e0d07db8b1c0839efb7bb: Status 404 returned error can't find the container with id fc0dd3ce23815f3f2411d0cb1c54a63946d45fb87b8e0d07db8b1c0839efb7bb
Oct 30 14:13:09.338365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:09.338338    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"]
Oct 30 14:13:09.460969 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:09.460936    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4" event={"ID":"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a","Type":"ContainerStarted","Data":"fc0dd3ce23815f3f2411d0cb1c54a63946d45fb87b8e0d07db8b1c0839efb7bb"}
Oct 30 14:13:09.464096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:09.464072    4530 patch_prober.go:29] interesting pod/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 14:13:09.464216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:09.464116    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 14:13:10.151836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:10.151747    4530 patch_prober.go:29] interesting pod/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 14:13:10.151836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:10.151806    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 14:13:10.463799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:10.463711    4530 patch_prober.go:29] interesting pod/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 14:13:10.463799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:10.463771    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 14:13:10.472473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:10.472400    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4" event={"ID":"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a","Type":"ContainerStarted","Data":"d0e82a0271c8c53e2cc1664c41fc56d77fd580b945a5d9dcc2ed3b74dd7d570b"}
Oct 30 14:13:10.544992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:10.544941    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4" podStartSLOduration=2.544926709 podStartE2EDuration="2.544926709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:10.541854324 +0000 UTC m=+1455.573753263" watchObservedRunningTime="2026-10-30 14:13:10.544926709 +0000 UTC m=+1455.576825648"
Oct 30 14:13:11.152154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:11.152123    4530 patch_prober.go:29] interesting pod/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 14:13:11.152339 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:11.152174    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 14:13:11.369227 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:13:11.369192    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:13:11.464174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:11.464017    4530 patch_prober.go:29] interesting pod/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 14:13:11.464174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:11.464070    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 14:13:11.527605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:11.527564    4530 generic.go:356] "Generic (PLEG): container finished" podID="c88ca0ca-4b55-4052-be9e-ed18a12a9d3a" containerID="d0e82a0271c8c53e2cc1664c41fc56d77fd580b945a5d9dcc2ed3b74dd7d570b" exitCode=0
Oct 30 14:13:11.527784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:11.527662    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4" event={"ID":"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a","Type":"ContainerDied","Data":"d0e82a0271c8c53e2cc1664c41fc56d77fd580b945a5d9dcc2ed3b74dd7d570b"}
Oct 30 14:13:12.157174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.157143    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:13:12.157594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.157575    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:13:12.158373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.158353    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:13:12.463332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.463251    4530 patch_prober.go:29] interesting pod/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 14:13:12.463332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.463304    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 14:13:12.665704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.665674    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"]
Oct 30 14:13:12.670940 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.670917    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:12.724653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.724623    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"]
Oct 30 14:13:12.752624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.752592    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9765472-baed-43a7-8370-63376548a4c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv\" (UID: \"d9765472-baed-43a7-8370-63376548a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:12.752781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.752656    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5kps\" (UniqueName: \"kubernetes.io/projected/d9765472-baed-43a7-8370-63376548a4c7-kube-api-access-x5kps\") pod \"maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv\" (UID: \"d9765472-baed-43a7-8370-63376548a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:12.752781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.752734    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9765472-baed-43a7-8370-63376548a4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv\" (UID: \"d9765472-baed-43a7-8370-63376548a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:12.755778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.755752    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:12.853978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.853940    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-openshift-ca\") pod \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\" (UID: \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\") "
Oct 30 14:13:12.854173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.854084    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-xks-ca\") pod \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\" (UID: \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\") "
Oct 30 14:13:12.854247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.854180    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-kube-api-access-wsg7v\" (UniqueName: \"kubernetes.io/projected/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-kube-api-access-wsg7v\") pod \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\" (UID: \"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a\") "
Oct 30 14:13:12.854417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.854388    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-openshift-ca" pod "c88ca0ca-4b55-4052-be9e-ed18a12a9d3a" (UID: "c88ca0ca-4b55-4052-be9e-ed18a12a9d3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:12.854677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.854654    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9765472-baed-43a7-8370-63376548a4c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv\" (UID: \"d9765472-baed-43a7-8370-63376548a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:12.854777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.854703    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5kps\" (UniqueName: \"kubernetes.io/projected/d9765472-baed-43a7-8370-63376548a4c7-kube-api-access-x5kps\") pod \"maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv\" (UID: \"d9765472-baed-43a7-8370-63376548a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:12.854777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.854740    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9765472-baed-43a7-8370-63376548a4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv\" (UID: \"d9765472-baed-43a7-8370-63376548a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:12.854991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.854971    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:12.855728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.855702    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9765472-baed-43a7-8370-63376548a4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv\" (UID: \"d9765472-baed-43a7-8370-63376548a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:12.857232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.857203    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-kube-api-access-wsg7v" pod "c88ca0ca-4b55-4052-be9e-ed18a12a9d3a" (UID: "c88ca0ca-4b55-4052-be9e-ed18a12a9d3a"). InnerVolumeSpecName "kube-api-access-wsg7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:12.857465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.857442    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-xks-ca" pod "c88ca0ca-4b55-4052-be9e-ed18a12a9d3a" (UID: "c88ca0ca-4b55-4052-be9e-ed18a12a9d3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:12.859051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.859032    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9765472-baed-43a7-8370-63376548a4c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv\" (UID: \"d9765472-baed-43a7-8370-63376548a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:12.907576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.906144    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5kps\" (UniqueName: \"kubernetes.io/projected/d9765472-baed-43a7-8370-63376548a4c7-kube-api-access-x5kps\") pod \"maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv\" (UID: \"d9765472-baed-43a7-8370-63376548a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:12.956447 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.956414    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:12.956447 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:12.956449    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsg7v\" (UniqueName: \"kubernetes.io/projected/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a-kube-api-access-wsg7v\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:13.019965 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:13.019888    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:13.341331 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:13:13.341300    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9765472_baed_43a7_8370_63376548a4c7.slice/crio-4b2cf05ef7fc24d52b9c5f72a29a23126ebb6312c59175442ce6f80e3083d4af WatchSource:0}: Error finding container 4b2cf05ef7fc24d52b9c5f72a29a23126ebb6312c59175442ce6f80e3083d4af: Status 404 returned error can't find the container with id 4b2cf05ef7fc24d52b9c5f72a29a23126ebb6312c59175442ce6f80e3083d4af
Oct 30 14:13:13.345786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:13.345760    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"]
Oct 30 14:13:13.463840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:13.463799    4530 patch_prober.go:29] interesting pod/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 14:13:13.464004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:13.463863    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 14:13:13.548918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:13.548886    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"
Oct 30 14:13:13.548918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:13.548907    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4" event={"ID":"c88ca0ca-4b55-4052-be9e-ed18a12a9d3a","Type":"ContainerDied","Data":"fc0dd3ce23815f3f2411d0cb1c54a63946d45fb87b8e0d07db8b1c0839efb7bb"}
Oct 30 14:13:13.549180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:13.548966    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc0dd3ce23815f3f2411d0cb1c54a63946d45fb87b8e0d07db8b1c0839efb7bb"
Oct 30 14:13:13.550593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:13.550565    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv" event={"ID":"d9765472-baed-43a7-8370-63376548a4c7","Type":"ContainerStarted","Data":"4b2cf05ef7fc24d52b9c5f72a29a23126ebb6312c59175442ce6f80e3083d4af"}
Oct 30 14:13:13.552153 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:13.552132    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"]
Oct 30 14:13:13.610680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:13.610651    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"]
Oct 30 14:13:14.488646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:14.488124    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:13:14.493752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:14.489329    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:13:14.493752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:14.491497    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:13:14.569347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:14.569245    4530 generic.go:356] "Generic (PLEG): container finished" podID="d9765472-baed-43a7-8370-63376548a4c7" containerID="6274f79be39d43d60abb3c2917d881baeb61192be1fdd2a9930960d60e50857c" exitCode=0
Oct 30 14:13:14.570521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:14.570487    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv" event={"ID":"d9765472-baed-43a7-8370-63376548a4c7","Type":"ContainerDied","Data":"6274f79be39d43d60abb3c2917d881baeb61192be1fdd2a9930960d60e50857c"}
Oct 30 14:13:15.750888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:15.750863    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:15.815107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:15.815073    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9765472-baed-43a7-8370-63376548a4c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9765472-baed-43a7-8370-63376548a4c7-maas-api-xks-ca\") pod \"d9765472-baed-43a7-8370-63376548a4c7\" (UID: \"d9765472-baed-43a7-8370-63376548a4c7\") "
Oct 30 14:13:15.815318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:15.815133    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9765472-baed-43a7-8370-63376548a4c7-kube-api-access-x5kps\" (UniqueName: \"kubernetes.io/projected/d9765472-baed-43a7-8370-63376548a4c7-kube-api-access-x5kps\") pod \"d9765472-baed-43a7-8370-63376548a4c7\" (UID: \"d9765472-baed-43a7-8370-63376548a4c7\") "
Oct 30 14:13:15.815318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:15.815214    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9765472-baed-43a7-8370-63376548a4c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9765472-baed-43a7-8370-63376548a4c7-maas-api-openshift-ca\") pod \"d9765472-baed-43a7-8370-63376548a4c7\" (UID: \"d9765472-baed-43a7-8370-63376548a4c7\") "
Oct 30 14:13:15.815855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:15.815824    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9765472-baed-43a7-8370-63376548a4c7-maas-api-openshift-ca" pod "d9765472-baed-43a7-8370-63376548a4c7" (UID: "d9765472-baed-43a7-8370-63376548a4c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:15.817802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:15.817773    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9765472-baed-43a7-8370-63376548a4c7-maas-api-xks-ca" pod "d9765472-baed-43a7-8370-63376548a4c7" (UID: "d9765472-baed-43a7-8370-63376548a4c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:15.818050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:15.818031    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9765472-baed-43a7-8370-63376548a4c7-kube-api-access-x5kps" pod "d9765472-baed-43a7-8370-63376548a4c7" (UID: "d9765472-baed-43a7-8370-63376548a4c7"). InnerVolumeSpecName "kube-api-access-x5kps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:15.917983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:15.917939    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9765472-baed-43a7-8370-63376548a4c7-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:15.917983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:15.917985    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5kps\" (UniqueName: \"kubernetes.io/projected/d9765472-baed-43a7-8370-63376548a4c7-kube-api-access-x5kps\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:15.918227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:15.918001    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9765472-baed-43a7-8370-63376548a4c7-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:16.588015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:16.587987    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"
Oct 30 14:13:16.588183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:16.588017    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv" event={"ID":"d9765472-baed-43a7-8370-63376548a4c7","Type":"ContainerDied","Data":"4b2cf05ef7fc24d52b9c5f72a29a23126ebb6312c59175442ce6f80e3083d4af"}
Oct 30 14:13:16.588183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:16.588057    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b2cf05ef7fc24d52b9c5f72a29a23126ebb6312c59175442ce6f80e3083d4af"
Oct 30 14:13:18.553426 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:18.553366    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7" podUID="6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7" containerName="payload-processing" containerID="cri-o://c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d" gracePeriod=25
Oct 30 14:13:18.611148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:18.611104    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq" podUID="1cb9017b-82b2-4962-8a4f-c70e795b19bc" containerName="payload-pre-processing" containerID="cri-o://454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c" gracePeriod=25
Oct 30 14:13:18.910808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:18.910780    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"
Oct 30 14:13:18.942045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:18.942021    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"
Oct 30 14:13:19.001185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.001005    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-kube-api-access-lndb2\" (UniqueName: \"kubernetes.io/projected/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-kube-api-access-lndb2\") pod \"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7\" (UID: \"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7\") "
Oct 30 14:13:19.001657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.001625    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-plugins-config-volume\") pod \"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7\" (UID: \"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7\") "
Oct 30 14:13:19.003389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.002090    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-plugins-config-volume" pod "6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7" (UID: "6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:19.003504 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.003467    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:19.004535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.004169    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-kube-api-access-lndb2" pod "6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7" (UID: "6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7"). InnerVolumeSpecName "kube-api-access-lndb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:19.104462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.104388    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb9017b-82b2-4962-8a4f-c70e795b19bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb9017b-82b2-4962-8a4f-c70e795b19bc-plugins-config-volume\") pod \"1cb9017b-82b2-4962-8a4f-c70e795b19bc\" (UID: \"1cb9017b-82b2-4962-8a4f-c70e795b19bc\") "
Oct 30 14:13:19.104607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.104592    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb9017b-82b2-4962-8a4f-c70e795b19bc-kube-api-access-kdc8c\" (UniqueName: \"kubernetes.io/projected/1cb9017b-82b2-4962-8a4f-c70e795b19bc-kube-api-access-kdc8c\") pod \"1cb9017b-82b2-4962-8a4f-c70e795b19bc\" (UID: \"1cb9017b-82b2-4962-8a4f-c70e795b19bc\") "
Oct 30 14:13:19.104748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.104725    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb9017b-82b2-4962-8a4f-c70e795b19bc-plugins-config-volume" pod "1cb9017b-82b2-4962-8a4f-c70e795b19bc" (UID: "1cb9017b-82b2-4962-8a4f-c70e795b19bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:19.105174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.105158    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lndb2\" (UniqueName: \"kubernetes.io/projected/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7-kube-api-access-lndb2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:19.105217 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.105177    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb9017b-82b2-4962-8a4f-c70e795b19bc-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:19.106615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.106596    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb9017b-82b2-4962-8a4f-c70e795b19bc-kube-api-access-kdc8c" pod "1cb9017b-82b2-4962-8a4f-c70e795b19bc" (UID: "1cb9017b-82b2-4962-8a4f-c70e795b19bc"). InnerVolumeSpecName "kube-api-access-kdc8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:19.206033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.205999    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdc8c\" (UniqueName: \"kubernetes.io/projected/1cb9017b-82b2-4962-8a4f-c70e795b19bc-kube-api-access-kdc8c\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:19.498471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.498373    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"]
Oct 30 14:13:19.499406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.499380    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7" containerName="payload-processing"
Oct 30 14:13:19.499406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.499408    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7" containerName="payload-processing"
Oct 30 14:13:19.499585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.499436    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1cb9017b-82b2-4962-8a4f-c70e795b19bc" containerName="payload-pre-processing"
Oct 30 14:13:19.499585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.499445    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb9017b-82b2-4962-8a4f-c70e795b19bc" containerName="payload-pre-processing"
Oct 30 14:13:19.499585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.499462    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d9765472-baed-43a7-8370-63376548a4c7" containerName="revoke-keys"
Oct 30 14:13:19.499585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.499469    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9765472-baed-43a7-8370-63376548a4c7" containerName="revoke-keys"
Oct 30 14:13:19.499585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.499523    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c88ca0ca-4b55-4052-be9e-ed18a12a9d3a" containerName="revoke-keys"
Oct 30 14:13:19.499585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.499532    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88ca0ca-4b55-4052-be9e-ed18a12a9d3a" containerName="revoke-keys"
Oct 30 14:13:19.499846 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.499700    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88ca0ca-4b55-4052-be9e-ed18a12a9d3a" containerName="revoke-keys"
Oct 30 14:13:19.499846 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.499718    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7" containerName="payload-processing"
Oct 30 14:13:19.499846 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.499733    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb9017b-82b2-4962-8a4f-c70e795b19bc" containerName="payload-pre-processing"
Oct 30 14:13:19.499846 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.499742    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9765472-baed-43a7-8370-63376548a4c7" containerName="revoke-keys"
Oct 30 14:13:19.503482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.503466    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.514929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.514906    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"]
Oct 30 14:13:19.519328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.519308    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.520792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.520760    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"]
Oct 30 14:13:19.525672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.525649    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5drtm\""
Oct 30 14:13:19.526127 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.526108    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.533439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.533412    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"]
Oct 30 14:13:19.558622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.557645    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"]
Oct 30 14:13:19.568084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.567518    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.571687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.571657    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"]
Oct 30 14:13:19.578243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.578217    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"]
Oct 30 14:13:19.611451 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611422    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.611683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611483    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.611683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611614    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.611683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611656    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.611683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611684    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7188a495-aaac-4e93-bbe4-0730b1190562-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.611946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611746    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.611946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611792    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7188a495-aaac-4e93-bbe4-0730b1190562-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.611946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611845    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.611946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611870    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbtmf\" (UniqueName: \"kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-kube-api-access-wbtmf\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.611946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611901    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.611946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611920    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.612268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611959    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad83b9c7-59fd-40da-9726-98f8366e3163-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.612268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.611976    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.612268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612026    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.612268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612069    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.612268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612125    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.612268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612164    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c719d3-00ae-46f7-8364-6af8392425b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.612268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612192    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88hb4\" (UniqueName: \"kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-kube-api-access-88hb4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.612268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612207    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.612268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612222    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.612268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612244    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.612823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612275    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.612823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612293    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.612823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612308    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.612823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612333    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.612823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612352    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.612823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612417    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fnrt\" (UniqueName: \"kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-kube-api-access-6fnrt\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.612823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.612647    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"]
Oct 30 14:13:19.626064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.626039    4530 generic.go:356] "Generic (PLEG): container finished" podID="1cb9017b-82b2-4962-8a4f-c70e795b19bc" containerID="454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c" exitCode=0
Oct 30 14:13:19.626193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.626082    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq" event={"ID":"1cb9017b-82b2-4962-8a4f-c70e795b19bc","Type":"ContainerDied","Data":"454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c"}
Oct 30 14:13:19.626193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.626097    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"
Oct 30 14:13:19.626193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.626141    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq" event={"ID":"1cb9017b-82b2-4962-8a4f-c70e795b19bc","Type":"ContainerDied","Data":"2c0ee269264574988b96e603fbf6bde1d8f1eae897e5fa28499804af20c10008"}
Oct 30 14:13:19.626193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.626171    4530 scope.go:122] "RemoveContainer" containerID="454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c"
Oct 30 14:13:19.627911 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.627888    4530 generic.go:356] "Generic (PLEG): container finished" podID="6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7" containerID="c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d" exitCode=0
Oct 30 14:13:19.628031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.627916    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7" event={"ID":"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7","Type":"ContainerDied","Data":"c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d"}
Oct 30 14:13:19.628031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.627960    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"
Oct 30 14:13:19.628149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.627962    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7" event={"ID":"6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7","Type":"ContainerDied","Data":"03e2e7c37e54242e659aa0160e61faf9983fbf62478b7a1a9c2be493b41c517d"}
Oct 30 14:13:19.637772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.637753    4530 scope.go:122] "RemoveContainer" containerID="454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c"
Oct 30 14:13:19.638033 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:13:19.638007    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c\": container with ID starting with 454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c not found: ID does not exist" containerID="454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c"
Oct 30 14:13:19.638121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.638043    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c"} err="failed to get container status \"454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c\": rpc error: code = NotFound desc = could not find container \"454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c\": container with ID starting with 454cab9b087a9f784ab7b8d31c857ed9433b2584ac2da9f5aaa8c40b438cb11c not found: ID does not exist"
Oct 30 14:13:19.638121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.638063    4530 scope.go:122] "RemoveContainer" containerID="c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d"
Oct 30 14:13:19.648725 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.648662    4530 scope.go:122] "RemoveContainer" containerID="c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d"
Oct 30 14:13:19.648951 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:13:19.648931    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d\": container with ID starting with c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d not found: ID does not exist" containerID="c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d"
Oct 30 14:13:19.649014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.648960    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d"} err="failed to get container status \"c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d\": rpc error: code = NotFound desc = could not find container \"c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d\": container with ID starting with c1dd0ac411d3dbf0ae602d634b1df2ae396888f468412527dfa9e11e4b61e18d not found: ID does not exist"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.715817    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.715886    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.715921    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.715968    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.716053    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcb8t\" (UniqueName: \"kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-kube-api-access-vcb8t\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.716117    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.716186    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fnrt\" (UniqueName: \"kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-kube-api-access-6fnrt\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.716220    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.716300    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.716348    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.716392    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.731767    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.731818    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7188a495-aaac-4e93-bbe4-0730b1190562-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.731889    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.731940    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7188a495-aaac-4e93-bbe4-0730b1190562-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.732020    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.732085    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.732120    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbtmf\" (UniqueName: \"kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-kube-api-access-wbtmf\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.732194    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.732238    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.732271    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.732381    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad83b9c7-59fd-40da-9726-98f8366e3163-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.732411    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.732567    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.732600    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.732718    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.733033    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.733153    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c719d3-00ae-46f7-8364-6af8392425b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.733212    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.733245    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.733317    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88hb4\" (UniqueName: \"kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-kube-api-access-88hb4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.733372    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.733403    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.733454    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.733495    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.733701    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.736460    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.737275    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.737934    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.749255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.738714    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c719d3-00ae-46f7-8364-6af8392425b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.751580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.749941    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.751580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.750083    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.751580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.750616    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.751580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.750881    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.751580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.751195    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.751580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.751306    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.751580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.751533    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.759194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.752106    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad83b9c7-59fd-40da-9726-98f8366e3163-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.759194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.755050    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7188a495-aaac-4e93-bbe4-0730b1190562-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.773355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.766831    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.773355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.769255    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.773355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.769359    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.773355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.769372    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.773355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.769427    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7188a495-aaac-4e93-bbe4-0730b1190562-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.773355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.769925    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.773355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.772865    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.773355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.773176    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.795829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.795759    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbtmf\" (UniqueName: \"kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-kube-api-access-wbtmf\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.820630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.819331    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.828191 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.828160    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88hb4\" (UniqueName: \"kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-kube-api-access-88hb4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d2gh4\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.829064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.829014    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.830442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.830411    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"]
Oct 30 14:13:19.831267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.831248    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fnrt\" (UniqueName: \"kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-kube-api-access-6fnrt\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-swl4n\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:19.833129 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.833099    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6gclk\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.835081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.835057    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.836871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.835671    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.836871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.835811    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.836871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.835838    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.836871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.835869    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.836871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.835956    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcb8t\" (UniqueName: \"kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-kube-api-access-vcb8t\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.836871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.835993    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.836871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.836066    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.836871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.836125    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.838016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.837988    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.838125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.838046    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.838301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.838278    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.839641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.838973    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.839641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.839453    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.839798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.839664    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.842406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.842381    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.845084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.845064    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-da21809f-78d4d5887f-wpppq"]
Oct 30 14:13:19.846325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.846308    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:19.866915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.866889    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:19.879194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.879152    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.879298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.879217    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcb8t\" (UniqueName: \"kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-kube-api-access-vcb8t\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7mtkg\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.908924 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.908887    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:19.957130 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.957097    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"]
Oct 30 14:13:19.965555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:19.965010    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-da21809f-7c7c7f9b8b-kgnb7"]
Oct 30 14:13:20.119575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.119496    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:20.184837 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:13:20.184679    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c719d3_00ae_46f7_8364_6af8392425b5.slice/crio-e675691c1e79e4b8c9dbe8310f814c07203608eaa6b16c888537172dc1c35fa0 WatchSource:0}: Error finding container e675691c1e79e4b8c9dbe8310f814c07203608eaa6b16c888537172dc1c35fa0: Status 404 returned error can't find the container with id e675691c1e79e4b8c9dbe8310f814c07203608eaa6b16c888537172dc1c35fa0
Oct 30 14:13:20.188159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.188122    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:20.188297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.188193    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:20.188297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.188223    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:20.194045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.194021    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"]
Oct 30 14:13:20.201147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.201124    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"]
Oct 30 14:13:20.212305 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.212260    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:20.212416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.212347    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:20.212416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.212394    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:20.266080 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.266035    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:20.266184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.266138    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:20.266243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.266188    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:20.271644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.271557    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"]
Oct 30 14:13:20.446676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.446640    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"]
Oct 30 14:13:20.447438 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:13:20.447398    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7188a495_aaac_4e93_bbe4_0730b1190562.slice/crio-8b5ae598309853892009ce9430fefc9a49d54f70191a1a82b7f26f0fd2c60990 WatchSource:0}: Error finding container 8b5ae598309853892009ce9430fefc9a49d54f70191a1a82b7f26f0fd2c60990: Status 404 returned error can't find the container with id 8b5ae598309853892009ce9430fefc9a49d54f70191a1a82b7f26f0fd2c60990
Oct 30 14:13:20.451471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.450932    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:20.451471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.451015    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:20.451471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.451059    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:20.642744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.642704    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" event={"ID":"a8c719d3-00ae-46f7-8364-6af8392425b5","Type":"ContainerStarted","Data":"7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67"}
Oct 30 14:13:20.643218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.642764    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" event={"ID":"a8c719d3-00ae-46f7-8364-6af8392425b5","Type":"ContainerStarted","Data":"e675691c1e79e4b8c9dbe8310f814c07203608eaa6b16c888537172dc1c35fa0"}
Oct 30 14:13:20.646022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.645991    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" event={"ID":"ad83b9c7-59fd-40da-9726-98f8366e3163","Type":"ContainerStarted","Data":"ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c"}
Oct 30 14:13:20.646161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.646043    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" event={"ID":"ad83b9c7-59fd-40da-9726-98f8366e3163","Type":"ContainerStarted","Data":"7aa7c04779dea7f48b407f9122760f768e339fbb186e6cc55615b0f102ab9c25"}
Oct 30 14:13:20.653417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.653374    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" event={"ID":"0f237ea7-c1f7-46ba-ad61-a5871dbf3049","Type":"ContainerStarted","Data":"7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524"}
Oct 30 14:13:20.653616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.653595    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" event={"ID":"0f237ea7-c1f7-46ba-ad61-a5871dbf3049","Type":"ContainerStarted","Data":"442d8d2699b8733fb352910ac4c3306cee230ea05e73cdb400fec1e7e6f839a9"}
Oct 30 14:13:20.657029 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.657005    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" event={"ID":"7188a495-aaac-4e93-bbe4-0730b1190562","Type":"ContainerStarted","Data":"11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045"}
Oct 30 14:13:20.657136 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.657050    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" event={"ID":"7188a495-aaac-4e93-bbe4-0730b1190562","Type":"ContainerStarted","Data":"8b5ae598309853892009ce9430fefc9a49d54f70191a1a82b7f26f0fd2c60990"}
Oct 30 14:13:20.763272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.763204    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podStartSLOduration=1.763182676 podStartE2EDuration="1.763182676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:20.70840612 +0000 UTC m=+1465.740305060" watchObservedRunningTime="2026-10-30 14:13:20.763182676 +0000 UTC m=+1465.795081616"
Oct 30 14:13:20.770065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.769977    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podStartSLOduration=1.769959097 podStartE2EDuration="1.769959097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:20.760424814 +0000 UTC m=+1465.792323756" watchObservedRunningTime="2026-10-30 14:13:20.769959097 +0000 UTC m=+1465.801858037"
Oct 30 14:13:20.824430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.824369    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podStartSLOduration=1.824351272 podStartE2EDuration="1.824351272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:20.81408473 +0000 UTC m=+1465.845983695" watchObservedRunningTime="2026-10-30 14:13:20.824351272 +0000 UTC m=+1465.856250216"
Oct 30 14:13:20.846950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.846907    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:20.866308 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.866245    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podStartSLOduration=1.8662266079999998 podStartE2EDuration="1.866226608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:20.864029658 +0000 UTC m=+1465.895928598" watchObservedRunningTime="2026-10-30 14:13:20.866226608 +0000 UTC m=+1465.898125549"
Oct 30 14:13:20.867106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.867083    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:20.909175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:20.909133    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:21.137640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:21.130011    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:21.137640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:21.130070    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:21.140313 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:21.140279    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb9017b-82b2-4962-8a4f-c70e795b19bc" path="/var/lib/kubelet/pods/1cb9017b-82b2-4962-8a4f-c70e795b19bc/volumes"
Oct 30 14:13:21.144554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:21.144511    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7" path="/var/lib/kubelet/pods/6d28cec1-fa9e-4614-b6d1-6f9fac5d5fc7/volumes"
Oct 30 14:13:21.145423 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:21.145390    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:21.235658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:21.235615    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:13:22.216573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.177305    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:22.216573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.177381    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:22.318282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.318244    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:22.318576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.318526    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:22.402573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.402053    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:22.402573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.402130    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:22.402573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.402245    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 14:13:22.402573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.402279    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 14:13:22.486630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.484921    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:22.486630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.485088    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp" podUID="ef6239cf-ec71-4299-8b6d-ef31d72c3162" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 30 14:13:22.896568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.855340    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:22.896568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.855402    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:22.896568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.875728    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 14:13:22.896568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.876451    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 14:13:22.896568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.876496    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 14:13:22.986305 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.986269    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:22.986605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:22.986576    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:23.056135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.056081    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:23.129957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.122111    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:23.149646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.144750    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:23.167631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.167589    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:23.196958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.196901    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:23.357815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.357781    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"]
Oct 30 14:13:23.358582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.358562    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da21809f-s8l4j"]
Oct 30 14:13:23.385576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.385500    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"]
Oct 30 14:13:23.404915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.404810    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy" containerID="cri-o://d9dad1866d2f6b0ee2758ae88cd7f686c33ebd7d600a11b0396558f6e45ac59a" gracePeriod=30
Oct 30 14:13:23.593571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.568640    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"]
Oct 30 14:13:23.700880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.700659    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:23.810463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.765040    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:13:23.810463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.765791    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:13:23.828026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.826833    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-25917498-jb2wv\" (UID: \"141128bf-6a3c-4894-b7d9-5502c5bcf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:23.828305 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.828280    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-25917498-jb2wv\" (UID: \"141128bf-6a3c-4894-b7d9-5502c5bcf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:23.829588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.828755    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c947d\" (UniqueName: \"kubernetes.io/projected/141128bf-6a3c-4894-b7d9-5502c5bcf339-kube-api-access-c947d\") pod \"maas-api-revoke-keys-e2e-ait-25917498-jb2wv\" (UID: \"141128bf-6a3c-4894-b7d9-5502c5bcf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:23.839645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.839610    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"]
Oct 30 14:13:23.869563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.848135    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:23.869563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.848192    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:23.890714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.890659    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:23.895412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.894019    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:13:23.895412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.893110    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 14:13:23.895412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.895049    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 14:13:23.911855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.910897    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:23.911855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.910958    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:23.932899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.932851    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-25917498-jb2wv\" (UID: \"141128bf-6a3c-4894-b7d9-5502c5bcf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:23.933602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.933569    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-25917498-jb2wv\" (UID: \"141128bf-6a3c-4894-b7d9-5502c5bcf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:23.934560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.934525    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c947d\" (UniqueName: \"kubernetes.io/projected/141128bf-6a3c-4894-b7d9-5502c5bcf339-kube-api-access-c947d\") pod \"maas-api-revoke-keys-e2e-ait-25917498-jb2wv\" (UID: \"141128bf-6a3c-4894-b7d9-5502c5bcf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:23.935049 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.934124    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-25917498-jb2wv\" (UID: \"141128bf-6a3c-4894-b7d9-5502c5bcf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:23.960218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:23.960105    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-25917498-jb2wv\" (UID: \"141128bf-6a3c-4894-b7d9-5502c5bcf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:24.017585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.004304    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c947d\" (UniqueName: \"kubernetes.io/projected/141128bf-6a3c-4894-b7d9-5502c5bcf339-kube-api-access-c947d\") pod \"maas-api-revoke-keys-e2e-ait-25917498-jb2wv\" (UID: \"141128bf-6a3c-4894-b7d9-5502c5bcf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:24.103652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.103310    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:24.132895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.130176    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:24.132895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.130240    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:24.201871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.201820    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:24.202339 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.202302    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:13:24.226647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.206986    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:24.253565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.227138    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:13:24.510974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.455434    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:13:24.869903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.855632    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:24.869903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.855691    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:24.969728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.969692    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:24.970004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.969975    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:24.971690 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.971665    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"]
Oct 30 14:13:24.997663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.990652    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 14:13:24.997663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:24.990862    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 14:13:25.282621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.281247    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:13:25.495564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.398719    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:25.495564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.399338    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded"
Oct 30 14:13:25.495564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.485850    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:25.495564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.489277    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-credential-socket\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.495564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.489397    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-podinfo\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.495564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.489567    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-envoy\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.503216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.503183    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee32a07-96da-46e2-86cc-b2deed6224fb-istiod-ca-cert\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.503496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.503472    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-token\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.503780 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.503760    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-socket\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.503923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.503908    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-data\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.504049 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.504034    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-certs\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.504165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.504150    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqv8c\" (UniqueName: \"kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-kube-api-access-xqv8c\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.641617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.639352    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-socket\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.641617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.639427    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-data\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.641617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.639468    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-certs\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.641617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.639511    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqv8c\" (UniqueName: \"kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-kube-api-access-xqv8c\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.641617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.639564    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-credential-socket\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.641617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.639665    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-podinfo\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.641617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.639819    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-envoy\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.641617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.639914    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee32a07-96da-46e2-86cc-b2deed6224fb-istiod-ca-cert\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.641617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.639965    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-token\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.641617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.641040    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-socket\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.641617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.641343    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-data\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.660814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.660331    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee32a07-96da-46e2-86cc-b2deed6224fb-istiod-ca-cert\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.661075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.661047    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-certs\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.661469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.661447    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-credential-socket\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:25.897626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:25.892773    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"]
Oct 30 14:13:26.020572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.017384    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-envoy\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:26.094702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.065261    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-podinfo\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:26.094702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.065771    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-token\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:26.094702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.067640    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:26.094702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.082683    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:26.094702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.082350    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:26.094702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.083116    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:26.124565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.118046    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:26.124565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.118103    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:26.124565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.118233    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 14:13:26.124565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.118262    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 14:13:26.200631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.175642    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:26.272567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.268580    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqv8c\" (UniqueName: \"kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-kube-api-access-xqv8c\") pod \"e2e-ait-4522b19c-openshift-default-b9d85f877-wj464\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") " pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:26.272567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.270661    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4522b19c-openshift-default-dockercfg-l2phj\""
Oct 30 14:13:26.558111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.539908    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:26.558111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.539989    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:26.771590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.755010    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795be2b8-287e-4fc5-b101-0c10cc5dcdad" path="/var/lib/kubelet/pods/795be2b8-287e-4fc5-b101-0c10cc5dcdad/volumes"
Oct 30 14:13:26.902805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.902694    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:26.920688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:26.903380    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:13:27.082612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.062789    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:27.082612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.062994    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Oct 30 14:13:27.096880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.094127    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:27.096880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.094214    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:27.096880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.094442    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 14:13:27.096880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.094491    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 14:13:27.096880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.096468    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 14:13:27.096880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.096708    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:13:27.325565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.300608    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:27.325565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.300668    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:27.444565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.435033    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:27.444867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.444822    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:27.556040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.519620    4530 patch_prober.go:29] interesting pod/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:13:27.556040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.519697    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:13:27.952302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.935714    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:27.952302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.939523    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 14:13:27.952302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.939590    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 14:13:27.952302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.939676    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:27.952302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.939698    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:27.952302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:27.948805    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:28.025601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.022010    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:28.025601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.022087    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:28.082566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.078260    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:28.163978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.163932    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:28.281668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.279826    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:28.281668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.279884    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:28.376562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.376299    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": context deadline exceeded"
Oct 30 14:13:28.384654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.384615    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/health\": context deadline exceeded"
Oct 30 14:13:28.648583 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:13:28.647611    4530 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.527s"
Oct 30 14:13:28.648583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.647655    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"]
Oct 30 14:13:28.648583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.647746    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:13:28.648583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.647766    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv" event={"ID":"141128bf-6a3c-4894-b7d9-5502c5bcf339","Type":"ContainerStarted","Data":"05aabbff34ab6b869c1b8db0e279534fa5e4223164d4cb0fc4de063c8f88a69d"}
Oct 30 14:13:28.848572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.847968    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:28.848572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.848047    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:28.903852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.903819    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 14:13:28.908799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.905015    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 14:13:28.936134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.930465    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:28.936134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.930510    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:28.937731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:28.937600    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:29.078683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:29.078634    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:29.298206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:29.224062    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Oct 30 14:13:29.298462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:29.296146    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 14:13:29.507958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:29.507919    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:29.508199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:29.508173    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:29.707621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:29.705288    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:13:29.880801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:29.873595    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 14:13:29.880801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:29.873674    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 14:13:29.940349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:29.909298    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:29.940349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:29.909355    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:29.942585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:29.942532    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:29.942810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:29.942782    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:30.079711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.079667    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"]
Oct 30 14:13:30.151761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.151723    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:30.152044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.152013    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:30.417566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.382989    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:30.432564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.431261    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:30.647716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.647673    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"]
Oct 30 14:13:30.776817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.776769    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:30.880399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.880353    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:30.880720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.880664    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:30.885592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.884960    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 14:13:30.885592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.885019    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 14:13:30.941564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.938227    4530 generic.go:356] "Generic (PLEG): container finished" podID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerID="d9dad1866d2f6b0ee2758ae88cd7f686c33ebd7d600a11b0396558f6e45ac59a" exitCode=0
Oct 30 14:13:30.945997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.942033    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:30.946216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.946189    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:30.950564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.946526    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" event={"ID":"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf","Type":"ContainerDied","Data":"d9dad1866d2f6b0ee2758ae88cd7f686c33ebd7d600a11b0396558f6e45ac59a"}
Oct 30 14:13:30.950812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.950788    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8521543e-z5kdg"]
Oct 30 14:13:30.950942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:30.950930    4530 scope.go:122] "RemoveContainer" containerID="7b44104c4852e9462ffc11defe3c373e5d3f25ddb625ce02e0c0c9d00c485be0"
Oct 30 14:13:31.146723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:31.146689    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:31.147299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:31.147268    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:31.481722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:31.402646    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:31.481722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:31.465814    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:31.512624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:31.512558    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded"
Oct 30 14:13:31.803575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:31.803517    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:13:31.815569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:31.814200    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:31.815569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:31.814302    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:31.815569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:31.814343    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:31.921198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:31.921155    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:31.921519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:31.921486    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:32.171155 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.171115    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:32.171705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.171648    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:32.218199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.216939    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a53caf1-226d-40b7-be45-31c6d59795a0" path="/var/lib/kubelet/pods/2a53caf1-226d-40b7-be45-31c6d59795a0/volumes"
Oct 30 14:13:32.229895 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:13:32.229847    4530 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a77bcbc_6491_4e7f_9842_e4ea8968a1bf.slice/crio-d9dad1866d2f6b0ee2758ae88cd7f686c33ebd7d600a11b0396558f6e45ac59a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a77bcbc_6491_4e7f_9842_e4ea8968a1bf.slice/crio-conmon-d9dad1866d2f6b0ee2758ae88cd7f686c33ebd7d600a11b0396558f6e45ac59a.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 14:13:32.265658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.264757    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:32.270512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.270468    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 14:13:32.273966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.273889    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 14:13:32.305856 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:13:32.281484    4530 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.163s"
Oct 30 14:13:32.321347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.320101    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:13:32.321347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.320256    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:13:32.321347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.309955    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:32.321347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.320642    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:32.349935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.337851    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-envoy\") pod \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") "
Oct 30 14:13:32.358154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.358117    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-kube-api-access-n6lj2\" (UniqueName: \"kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-kube-api-access-n6lj2\") pod \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") "
Oct 30 14:13:32.361753 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.361716    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-data\") pod \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") "
Oct 30 14:13:32.379436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.379394    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-data" pod "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" (UID: "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:32.379682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.379659    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-certs\") pod \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") "
Oct 30 14:13:32.379905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.379887    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-socket\") pod \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") "
Oct 30 14:13:32.380285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.380259    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-token\" (UniqueName: \"kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-token\") pod \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") "
Oct 30 14:13:32.380590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.380519    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-podinfo\") pod \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") "
Oct 30 14:13:32.381840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.381819    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-credential-socket\") pod \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") "
Oct 30 14:13:32.382022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.381996    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istiod-ca-cert\") pod \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\" (UID: \"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf\") "
Oct 30 14:13:32.394205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.394180    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:13:32.418037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.407637    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-certs" pod "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" (UID: "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:32.420494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.420429    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istiod-ca-cert" pod "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" (UID: "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:32.422659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.422623    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-credential-socket" pod "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" (UID: "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:32.424955 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.424922    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-kube-api-access-n6lj2" pod "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" (UID: "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf"). InnerVolumeSpecName "kube-api-access-n6lj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:32.445408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.445365    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-socket" pod "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" (UID: "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:32.467617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.467592    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6lj2\" (UniqueName: \"kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-kube-api-access-n6lj2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.467783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.467760    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.467949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.467934    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.468068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.468055    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.468191 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.468176    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.504250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.504215    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" event={"ID":"9ee32a07-96da-46e2-86cc-b2deed6224fb","Type":"ContainerStarted","Data":"c45000f617d9e4ee2effd35f070fe24560049f598f3e0358e3bb2a7fb1532305"}
Oct 30 14:13:32.506714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.506686    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-envoy" pod "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" (UID: "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:32.542884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.542832    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:13:32.543756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.543718    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5" event={"ID":"4a77bcbc-6491-4e7f-9842-e4ea8968a1bf","Type":"ContainerDied","Data":"8aac426dc443868f44acb20fa60788e0a046eb5f6ebb48b2797bf93f974671ae"}
Oct 30 14:13:32.543999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.543979    4530 scope.go:122] "RemoveContainer" containerID="d9dad1866d2f6b0ee2758ae88cd7f686c33ebd7d600a11b0396558f6e45ac59a"
Oct 30 14:13:32.548666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.545158    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"
Oct 30 14:13:32.548666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.547636    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-podinfo" pod "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" (UID: "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:13:32.548666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.547787    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-token" pod "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" (UID: "4a77bcbc-6491-4e7f-9842-e4ea8968a1bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:32.574415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.574365    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.574690 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.574669    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.574909 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.574890    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.575083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.575066    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.597588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.597524    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"]
Oct 30 14:13:32.611652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.611622    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d08a9267-9btxg"]
Oct 30 14:13:32.617866 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.617834    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"]
Oct 30 14:13:32.619249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.619223    4530 cpu_manager.go:414] "containerMap: removing container" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy"
Oct 30 14:13:32.619402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.619387    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy"
Oct 30 14:13:32.619514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.619502    4530 cpu_manager.go:414] "containerMap: removing container" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy"
Oct 30 14:13:32.619637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.619623    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy"
Oct 30 14:13:32.619926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.619908    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy"
Oct 30 14:13:32.620039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.620027    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" containerName="istio-proxy"
Oct 30 14:13:32.643704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.642526    4530 generic.go:356] "Generic (PLEG): container finished" podID="141128bf-6a3c-4894-b7d9-5502c5bcf339" containerID="838f41ea41840ca566f9c48501d2cebc892f3ccc6fa4cbe7ea1e20cf61d89510" exitCode=0
Oct 30 14:13:32.679045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.678900    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv" podStartSLOduration=9.678881414 podStartE2EDuration="9.678881414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:31.654949068 +0000 UTC m=+1476.686848009" watchObservedRunningTime="2026-10-30 14:13:32.678881414 +0000 UTC m=+1477.710780355"
Oct 30 14:13:32.728527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.728491    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv" event={"ID":"141128bf-6a3c-4894-b7d9-5502c5bcf339","Type":"ContainerDied","Data":"838f41ea41840ca566f9c48501d2cebc892f3ccc6fa4cbe7ea1e20cf61d89510"}
Oct 30 14:13:32.728844 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.728807    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"]
Oct 30 14:13:32.729027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.729007    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"]
Oct 30 14:13:32.729273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.729254    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:32.777323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.777291    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-75443e-openshift-default-dockercfg-r7cs9\""
Oct 30 14:13:32.847552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.847507    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:32.847718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.847572    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:32.863952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.863885    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"]
Oct 30 14:13:32.868531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.867728    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 14:13:32.868531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.867777    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 14:13:32.875057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.875024    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-da21809f-gw-openshift-default-654f97494f-h8rc5"]
Oct 30 14:13:32.905303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.905228    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-podinfo\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:32.905629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.905583    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-credential-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:32.906624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.906043    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istiod-ca-cert\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:32.906624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.906146    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-data\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:32.906624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.906179    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw29m\" (UniqueName: \"kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-kube-api-access-fw29m\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:32.906624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.906258    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-certs\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:32.906624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.906335    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:32.906624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.906377    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-token\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:32.906624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.906408    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-envoy\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:32.909526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.909420    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:32.909526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:32.909471    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:33.008905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.008340    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.008905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.008426    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-token\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.008905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.008458    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-envoy\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.008905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.008594    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-podinfo\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.008905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.008638    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-credential-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.008905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.008874    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istiod-ca-cert\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.009369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.009322    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-data\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.009438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.009375    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw29m\" (UniqueName: \"kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-kube-api-access-fw29m\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.009504 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.009460    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-certs\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.009977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.009579    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-credential-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.009977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.009904    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-certs\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.010174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.010158    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-data\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.011137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.011052    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istiod-ca-cert\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.011574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.011526    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.016576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.015968    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-podinfo\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.021757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.021711    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-envoy\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.036501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.036468    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-token\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.037410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.037382    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw29m\" (UniqueName: \"kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-kube-api-access-fw29m\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.099411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.099373    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:33.120465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.120428    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:33.120673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.120491    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:33.137416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.137372    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a77bcbc-6491-4e7f-9842-e4ea8968a1bf" path="/var/lib/kubelet/pods/4a77bcbc-6491-4e7f-9842-e4ea8968a1bf/volumes"
Oct 30 14:13:33.138369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.138345    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd8ceea-0d38-4570-bab7-0cdd57f8723a" path="/var/lib/kubelet/pods/fcd8ceea-0d38-4570-bab7-0cdd57f8723a/volumes"
Oct 30 14:13:33.488309 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.488272    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"]
Oct 30 14:13:33.506254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.506198    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:33.506391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.506299    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:33.506391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.506346    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:33.664571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.664176    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" event={"ID":"b7bf62ef-287a-4ede-95e9-a0c8d8354141","Type":"ContainerStarted","Data":"b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167"}
Oct 30 14:13:33.664571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.664249    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" event={"ID":"b7bf62ef-287a-4ede-95e9-a0c8d8354141","Type":"ContainerStarted","Data":"9826a92c562fd48e150b265cd6dde355dbfc1611e43ac4237bad02a5917fbecb"}
Oct 30 14:13:33.668033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.668000    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" event={"ID":"9ee32a07-96da-46e2-86cc-b2deed6224fb","Type":"ContainerStarted","Data":"6fbbe14c01ca93a04cc52a549e99536db241ebf2ff9299ea4ba9b4e92379482b"}
Oct 30 14:13:33.674680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.674621    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2" podUID="dfa59e71-0a8c-4e33-8076-7e6a72fd4f30" containerName="payload-processing" containerID="cri-o://2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a" gracePeriod=25
Oct 30 14:13:33.763433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.763324    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" podStartSLOduration=2.7633033190000003 podStartE2EDuration="2.763303319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:33.752982404 +0000 UTC m=+1478.784881344" watchObservedRunningTime="2026-10-30 14:13:33.763303319 +0000 UTC m=+1478.795202259"
Oct 30 14:13:33.851592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.850332    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 14:13:33.851592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.850387    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 14:13:33.890565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.877300    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" podStartSLOduration=9.877270439 podStartE2EDuration="9.877270439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:33.813349748 +0000 UTC m=+1478.845248711" watchObservedRunningTime="2026-10-30 14:13:33.877270439 +0000 UTC m=+1478.909169379"
Oct 30 14:13:33.969573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.967157    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 14:13:33.969573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:33.967255    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 14:13:34.009564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.008286    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:34.112872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.018862    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:34.112872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.107334    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:34.152590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.150968    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 14:13:34.152590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.151031    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 14:13:34.152590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.151121    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 14:13:34.152590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.151153    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 14:13:34.370127 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:13:34.370052    4530 logging.go:55] [core] [Channel #1238 SubChannel #1239]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.133:50052", ServerName: "10.133.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.133:50052: connect: connection refused"
Oct 30 14:13:34.393705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.392599    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:34.510335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.510298    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-xks-ca\") pod \"141128bf-6a3c-4894-b7d9-5502c5bcf339\" (UID: \"141128bf-6a3c-4894-b7d9-5502c5bcf339\") "
Oct 30 14:13:34.524368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.524336    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141128bf-6a3c-4894-b7d9-5502c5bcf339-kube-api-access-c947d\" (UniqueName: \"kubernetes.io/projected/141128bf-6a3c-4894-b7d9-5502c5bcf339-kube-api-access-c947d\") pod \"141128bf-6a3c-4894-b7d9-5502c5bcf339\" (UID: \"141128bf-6a3c-4894-b7d9-5502c5bcf339\") "
Oct 30 14:13:34.524635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.524614    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-openshift-ca\") pod \"141128bf-6a3c-4894-b7d9-5502c5bcf339\" (UID: \"141128bf-6a3c-4894-b7d9-5502c5bcf339\") "
Oct 30 14:13:34.575120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.574869    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-openshift-ca" pod "141128bf-6a3c-4894-b7d9-5502c5bcf339" (UID: "141128bf-6a3c-4894-b7d9-5502c5bcf339"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:34.581287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.581255    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141128bf-6a3c-4894-b7d9-5502c5bcf339-kube-api-access-c947d" pod "141128bf-6a3c-4894-b7d9-5502c5bcf339" (UID: "141128bf-6a3c-4894-b7d9-5502c5bcf339"). InnerVolumeSpecName "kube-api-access-c947d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:34.591274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.591225    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-xks-ca" pod "141128bf-6a3c-4894-b7d9-5502c5bcf339" (UID: "141128bf-6a3c-4894-b7d9-5502c5bcf339"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:34.637125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.637042    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:34.642656 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.642629    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c947d\" (UniqueName: \"kubernetes.io/projected/141128bf-6a3c-4894-b7d9-5502c5bcf339-kube-api-access-c947d\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:34.642835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.642818    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/141128bf-6a3c-4894-b7d9-5502c5bcf339-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:34.652939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.652863    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"
Oct 30 14:13:34.699606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.699024    4530 generic.go:356] "Generic (PLEG): container finished" podID="dfa59e71-0a8c-4e33-8076-7e6a72fd4f30" containerID="2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a" exitCode=0
Oct 30 14:13:34.699606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.699153    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2" event={"ID":"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30","Type":"ContainerDied","Data":"2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a"}
Oct 30 14:13:34.699606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.699239    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2" event={"ID":"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30","Type":"ContainerDied","Data":"a641f1670713ef3613aa456179d04d705dc6e454a775f6d6840ca7d9e577e948"}
Oct 30 14:13:34.699606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.699275    4530 scope.go:122] "RemoveContainer" containerID="2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a"
Oct 30 14:13:34.699606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.699423    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"
Oct 30 14:13:34.721580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.712729    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"
Oct 30 14:13:34.722483 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.722450    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv" event={"ID":"141128bf-6a3c-4894-b7d9-5502c5bcf339","Type":"ContainerDied","Data":"05aabbff34ab6b869c1b8db0e279534fa5e4223164d4cb0fc4de063c8f88a69d"}
Oct 30 14:13:34.722695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.722675    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05aabbff34ab6b869c1b8db0e279534fa5e4223164d4cb0fc4de063c8f88a69d"
Oct 30 14:13:34.724133 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.724114    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:34.726238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.726219    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:13:34.748044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.748017    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-kube-api-access-zz86r\" (UniqueName: \"kubernetes.io/projected/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-kube-api-access-zz86r\") pod \"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30\" (UID: \"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30\") "
Oct 30 14:13:34.748354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.748336    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-plugins-config-volume\") pod \"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30\" (UID: \"dfa59e71-0a8c-4e33-8076-7e6a72fd4f30\") "
Oct 30 14:13:34.749810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.749784    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-plugins-config-volume" pod "dfa59e71-0a8c-4e33-8076-7e6a72fd4f30" (UID: "dfa59e71-0a8c-4e33-8076-7e6a72fd4f30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:34.755682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.755509    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:34.871603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.826657    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-kube-api-access-zz86r" pod "dfa59e71-0a8c-4e33-8076-7e6a72fd4f30" (UID: "dfa59e71-0a8c-4e33-8076-7e6a72fd4f30"). InnerVolumeSpecName "kube-api-access-zz86r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:34.895823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.853377    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:34.895823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.872808    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:34.895823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.884643    4530 scope.go:122] "RemoveContainer" containerID="2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a"
Oct 30 14:13:34.895823 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:13:34.885064    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a\": container with ID starting with 2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a not found: ID does not exist" containerID="2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a"
Oct 30 14:13:34.895823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.885105    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a"} err="failed to get container status \"2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a\": rpc error: code = NotFound desc = could not find container \"2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a\": container with ID starting with 2bfd640e3ea63a012918be6ad1650c30f3b6de118b1d264fa6957a6c5cc0c52a not found: ID does not exist"
Oct 30 14:13:34.912688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.912664    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:34.943912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.942127    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:13:34.943912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:34.942190    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:35.121559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.121512    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz86r\" (UniqueName: \"kubernetes.io/projected/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30-kube-api-access-zz86r\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:35.323525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.323486    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2" podUID="dfa59e71-0a8c-4e33-8076-7e6a72fd4f30" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.133:50052\" within 1s: context deadline exceeded"
Oct 30 14:13:35.350829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.350801    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:35.351103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.351077    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:35.407955 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.407922    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"]
Oct 30 14:13:35.419894 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.419855    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ea3e7b1d-77f5ddd568-r9gb2"]
Oct 30 14:13:35.478564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.469277    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:35.726180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.725530    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:35.729510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.727490    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:13:35.729510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.729032    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:13:35.895621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.883448    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6gclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:13:35.895621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.883509    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:35.988641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.987067    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:13:35.988641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:35.987131    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:36.207267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:36.184308    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:13:36.207267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:36.184371    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:36.310676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:36.310645    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:36.328834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:36.328808    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:13:36.339932 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:36.339905    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:36.345212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:36.345185    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:13:36.589863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:36.589441    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:36.591726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:36.591697    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:13:37.123958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:37.123924    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa59e71-0a8c-4e33-8076-7e6a72fd4f30" path="/var/lib/kubelet/pods/dfa59e71-0a8c-4e33-8076-7e6a72fd4f30/volumes"
Oct 30 14:13:40.678093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.678013    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"]
Oct 30 14:13:40.678756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.678740    4530 cpu_manager.go:414] "containerMap: removing container" podUID="141128bf-6a3c-4894-b7d9-5502c5bcf339" containerName="revoke-keys"
Oct 30 14:13:40.678756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.678756    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141128bf-6a3c-4894-b7d9-5502c5bcf339" containerName="revoke-keys"
Oct 30 14:13:40.678881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.678770    4530 cpu_manager.go:414] "containerMap: removing container" podUID="dfa59e71-0a8c-4e33-8076-7e6a72fd4f30" containerName="payload-processing"
Oct 30 14:13:40.678881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.678775    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa59e71-0a8c-4e33-8076-7e6a72fd4f30" containerName="payload-processing"
Oct 30 14:13:40.678881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.678876    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="141128bf-6a3c-4894-b7d9-5502c5bcf339" containerName="revoke-keys"
Oct 30 14:13:40.678969 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.678886    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa59e71-0a8c-4e33-8076-7e6a72fd4f30" containerName="payload-processing"
Oct 30 14:13:40.684394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.684373    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.697140 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.697114    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"]
Oct 30 14:13:40.702796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.702776    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.705744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.705376    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"]
Oct 30 14:13:40.706875 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.706832    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-39529e-openshift-default-dockercfg-l4zq2\""
Oct 30 14:13:40.711433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.711410    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.714299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.714275    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"]
Oct 30 14:13:40.729212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.729187    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"]
Oct 30 14:13:40.736506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.736483    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"]
Oct 30 14:13:40.859343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859310    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6qhx\" (UniqueName: \"kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-kube-api-access-n6qhx\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.859501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859365    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.859501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859470    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.859623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859563    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.859674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859621    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.859674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859650    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.859752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859675    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.859752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859741    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.859838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859767    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.859871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859827    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.859911 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859875    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.859954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859917    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.859954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.859938    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.860056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860016    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec037157-c68e-45fe-ba80-7cf66725d2bf-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.860104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860057    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.860104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860074    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.860198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860118    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.860248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860222    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.860281 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860246    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.860329 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860282    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.860380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860351    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.860417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860396    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.860467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860422    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvkbb\" (UniqueName: \"kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-kube-api-access-rvkbb\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.860467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860440    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.860590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860483    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.860590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860531    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.860688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.860591    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rrt\" (UniqueName: \"kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-kube-api-access-t6rrt\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.962221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962158    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.962221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962188    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.962221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962210    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.962396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962321    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.962396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962362    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.962470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962404    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvkbb\" (UniqueName: \"kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-kube-api-access-rvkbb\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.962470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962436    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.962599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962466    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.962670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962624    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.962728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962690    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.962783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962726    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.962783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962529    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.962882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962797    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rrt\" (UniqueName: \"kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-kube-api-access-t6rrt\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.962954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.962938    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6qhx\" (UniqueName: \"kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-kube-api-access-n6qhx\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.963047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963004    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.963047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963019    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.963163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963079    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.963163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963157    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.963275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963208    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.963275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963243    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.963380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963291    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.963380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963320    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.963380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963337    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.963380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963369    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.963627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963382    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.963627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963422    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.963627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963449    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.963627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963485    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.963627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963514    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.963627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963585    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec037157-c68e-45fe-ba80-7cf66725d2bf-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.963941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963658    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.963941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963663    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.963941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963879    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.963941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963893    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.964154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963943    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.964154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.963957    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.964277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.964213    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.964879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.964488    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.964879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.964765    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.964879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.964826    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec037157-c68e-45fe-ba80-7cf66725d2bf-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.965097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.964990    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.965392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.965353    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.965496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.965429    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.965781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.965757    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.966240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.966217    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.966509 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.966487    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.966881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.966861    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.967106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.967088    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.989825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.989798    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.989972 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.989931    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.990055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.989984    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rrt\" (UniqueName: \"kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-kube-api-access-t6rrt\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:40.990124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.990042    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvkbb\" (UniqueName: \"kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-kube-api-access-rvkbb\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:40.990470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.990445    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6qhx\" (UniqueName: \"kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-kube-api-access-n6qhx\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.990798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.990781    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:40.999738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:40.999718    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:41.030612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.030588    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:41.044169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.043990    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:41.325610 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:13:41.325581    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4c0803_2f0b_4718_aca6_a4b43ae47c20.slice/crio-ee0ce7be2a2c00dde4c4ed8f847b1576d2143b3d94a1d4f346f471fcde4031da WatchSource:0}: Error finding container ee0ce7be2a2c00dde4c4ed8f847b1576d2143b3d94a1d4f346f471fcde4031da: Status 404 returned error can't find the container with id ee0ce7be2a2c00dde4c4ed8f847b1576d2143b3d94a1d4f346f471fcde4031da
Oct 30 14:13:41.328272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.328234    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:41.328392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.328320    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:41.328392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.328367    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:41.330021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.329997    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"]
Oct 30 14:13:41.365322 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:13:41.365291    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec037157_c68e_45fe_ba80_7cf66725d2bf.slice/crio-bff480bd09663c7c62681e8e75829ea11e2f6644fa6cbd28448757959716cbdf WatchSource:0}: Error finding container bff480bd09663c7c62681e8e75829ea11e2f6644fa6cbd28448757959716cbdf: Status 404 returned error can't find the container with id bff480bd09663c7c62681e8e75829ea11e2f6644fa6cbd28448757959716cbdf
Oct 30 14:13:41.368192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.368151    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:41.368298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.368278    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:41.368356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.368329    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:41.370835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.370810    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"]
Oct 30 14:13:41.376724 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:13:41.376698    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:13:41.397187 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:13:41.397155    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b7b8d4_21ed_441c_8a3a_92936fe9ab4c.slice/crio-afdf03e9562ab6d2a9e1ae7e4780f7b41f968baac9680b2aa9cca58cb851ded1 WatchSource:0}: Error finding container afdf03e9562ab6d2a9e1ae7e4780f7b41f968baac9680b2aa9cca58cb851ded1: Status 404 returned error can't find the container with id afdf03e9562ab6d2a9e1ae7e4780f7b41f968baac9680b2aa9cca58cb851ded1
Oct 30 14:13:41.401553 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.401492    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:41.401823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.401796    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:41.402060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.401996    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:13:41.408163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.408133    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"]
Oct 30 14:13:41.492804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.491173    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" event={"ID":"bb4c0803-2f0b-4718-aca6-a4b43ae47c20","Type":"ContainerStarted","Data":"c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3"}
Oct 30 14:13:41.492804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.491234    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" event={"ID":"bb4c0803-2f0b-4718-aca6-a4b43ae47c20","Type":"ContainerStarted","Data":"ee0ce7be2a2c00dde4c4ed8f847b1576d2143b3d94a1d4f346f471fcde4031da"}
Oct 30 14:13:41.497707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.495958    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" event={"ID":"ec037157-c68e-45fe-ba80-7cf66725d2bf","Type":"ContainerStarted","Data":"95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b"}
Oct 30 14:13:41.497707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.496010    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" event={"ID":"ec037157-c68e-45fe-ba80-7cf66725d2bf","Type":"ContainerStarted","Data":"bff480bd09663c7c62681e8e75829ea11e2f6644fa6cbd28448757959716cbdf"}
Oct 30 14:13:41.499590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.499565    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" event={"ID":"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c","Type":"ContainerStarted","Data":"afdf03e9562ab6d2a9e1ae7e4780f7b41f968baac9680b2aa9cca58cb851ded1"}
Oct 30 14:13:41.547601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:41.543501    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" podStartSLOduration=1.5434836870000002 podStartE2EDuration="1.543483687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:41.542235237 +0000 UTC m=+1486.574134179" watchObservedRunningTime="2026-10-30 14:13:41.543483687 +0000 UTC m=+1486.575382628"
Oct 30 14:13:42.008569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:42.007322    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:42.021969 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:42.019220    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 14:13:42.021969 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:42.019298    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 14:13:42.053585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:42.052742    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:42.053585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:42.052931    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 14:13:42.053585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:42.052974    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 14:13:42.504619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:42.499371    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" podStartSLOduration=2.499350374 podStartE2EDuration="2.499350374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:41.597659645 +0000 UTC m=+1486.629558585" watchObservedRunningTime="2026-10-30 14:13:42.499350374 +0000 UTC m=+1487.531249317"
Oct 30 14:13:42.560453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:42.548483    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"]
Oct 30 14:13:42.874566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:42.857965    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea3e7b1d-26qgv"]
Oct 30 14:13:43.038637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:42.975651    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"]
Oct 30 14:13:43.038637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:42.975891    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy" containerID="cri-o://50af43fa9ee96aa5787ac51ef5cc82b3d02cbbec5776ca1eda233af45848b6a3" gracePeriod=30
Oct 30 14:13:43.038637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:43.031669    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 14:13:43.038637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:43.031731    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 14:13:43.078571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:43.078349    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 14:13:43.078571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:43.078411    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 14:13:43.602566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:43.597117    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9765472-baed-43a7-8370-63376548a4c7" path="/var/lib/kubelet/pods/d9765472-baed-43a7-8370-63376548a4c7/volumes"
Oct 30 14:13:43.650534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:43.649638    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:43.650891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:43.650854    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" event={"ID":"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c","Type":"ContainerStarted","Data":"d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f"}
Oct 30 14:13:43.738565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:43.712886    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" podStartSLOduration=3.712865742 podStartE2EDuration="3.712865742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:43.710900901 +0000 UTC m=+1488.742799852" watchObservedRunningTime="2026-10-30 14:13:43.712865742 +0000 UTC m=+1488.744764679"
Oct 30 14:13:44.009050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:44.008950    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 14:13:44.009661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:44.009619    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 14:13:44.031340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:44.031297    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 14:13:44.031553 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:44.031360    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 14:13:44.050579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:44.049652    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:44.050579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:44.049781    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 14:13:44.050579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:44.049824    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" podUID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 14:13:44.733568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:44.713052    4530 patch_prober.go:29] interesting pod/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:13:44.733893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:44.733852    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:13:45.004620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:45.002972    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 14:13:45.004620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:45.003037    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 14:13:45.039500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:45.035741    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 14:13:45.039500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:45.035813    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 14:13:45.045923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:45.045883    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 14:13:45.046307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:45.046253    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" podUID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 14:13:45.967372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:45.967337    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"]
Oct 30 14:13:45.973346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:45.973324    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:46.011682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.011647    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 14:13:46.034069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.034034    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 14:13:46.067769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.027102    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:13:46.067997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.026990    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:13:46.068094 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.032088    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 14:13:46.068208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.068188    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 14:13:46.068311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.064898    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40d77425-xksls\" (UID: \"025875c5-a966-4f05-bf7f-b68c9ba00b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:46.069498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.068830    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv2c9\" (UniqueName: \"kubernetes.io/projected/025875c5-a966-4f05-bf7f-b68c9ba00b73-kube-api-access-qv2c9\") pod \"maas-api-revoke-sub-40d77425-xksls\" (UID: \"025875c5-a966-4f05-bf7f-b68c9ba00b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:46.069921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.069899    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40d77425-xksls\" (UID: \"025875c5-a966-4f05-bf7f-b68c9ba00b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:46.070117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.053943    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 14:13:46.070194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.070147    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" podUID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 14:13:46.070513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.070487    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"]
Oct 30 14:13:46.171827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.171770    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv2c9\" (UniqueName: \"kubernetes.io/projected/025875c5-a966-4f05-bf7f-b68c9ba00b73-kube-api-access-qv2c9\") pod \"maas-api-revoke-sub-40d77425-xksls\" (UID: \"025875c5-a966-4f05-bf7f-b68c9ba00b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:46.171996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.171944    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40d77425-xksls\" (UID: \"025875c5-a966-4f05-bf7f-b68c9ba00b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:46.172241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.172223    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40d77425-xksls\" (UID: \"025875c5-a966-4f05-bf7f-b68c9ba00b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:46.173125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.173081    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40d77425-xksls\" (UID: \"025875c5-a966-4f05-bf7f-b68c9ba00b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:46.178688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.177639    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40d77425-xksls\" (UID: \"025875c5-a966-4f05-bf7f-b68c9ba00b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:46.232624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.217389    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv2c9\" (UniqueName: \"kubernetes.io/projected/025875c5-a966-4f05-bf7f-b68c9ba00b73-kube-api-access-qv2c9\") pod \"maas-api-revoke-sub-40d77425-xksls\" (UID: \"025875c5-a966-4f05-bf7f-b68c9ba00b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:46.336852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:46.336813    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:47.174595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:47.174560    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 14:13:47.175009 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:47.174634    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 14:13:47.175198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:47.175158    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 14:13:47.175299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:47.175206    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" podUID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 14:13:47.175299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:47.175271    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 14:13:47.175299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:47.175292    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 14:13:47.338714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:47.338649    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"]
Oct 30 14:13:48.010999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:48.010959    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 14:13:48.011286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:48.011249    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 14:13:48.034566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:48.032515    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 14:13:48.034566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:48.032603    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 14:13:48.056527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:48.046990    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 14:13:48.056527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:48.047038    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" podUID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 14:13:49.212580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.212525    4530 generic.go:356] "Generic (PLEG): container finished" podID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerID="50af43fa9ee96aa5787ac51ef5cc82b3d02cbbec5776ca1eda233af45848b6a3" exitCode=0
Oct 30 14:13:49.239246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.239208    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 14:13:49.239528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.239495    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 14:13:49.240055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.240024    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 14:13:49.240251 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.240228    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" podUID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 14:13:49.240426 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.240399    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 14:13:49.240507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.240445    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 14:13:49.271565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.267976    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:49.299434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.299395    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" event={"ID":"8e9979cc-ae59-4c43-abfe-88937c76fb2b","Type":"ContainerDied","Data":"50af43fa9ee96aa5787ac51ef5cc82b3d02cbbec5776ca1eda233af45848b6a3"}
Oct 30 14:13:49.357781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.357741    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls" event={"ID":"025875c5-a966-4f05-bf7f-b68c9ba00b73","Type":"ContainerStarted","Data":"b6ae0d4bc6e27062b3f46b7c43e0b9c43c062a024bd51d4ae92fd7398323909e"}
Oct 30 14:13:49.358032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.357991    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls" event={"ID":"025875c5-a966-4f05-bf7f-b68c9ba00b73","Type":"ContainerStarted","Data":"60a86378234ecdc06e04f2813e47a00dce4194762ac4d86e3d0f1a7e5aac22c3"}
Oct 30 14:13:49.358235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.352621    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:13:49.359340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.359319    4530 scope.go:122] "RemoveContainer" containerID="0412c56bf8fc0e24127c45de69f867feb24b8fb81760376a89bdaa56aae15f5c"
Oct 30 14:13:49.473929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.473854    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls" podStartSLOduration=4.473835275 podStartE2EDuration="4.473835275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:49.423787737 +0000 UTC m=+1494.455686677" watchObservedRunningTime="2026-10-30 14:13:49.473835275 +0000 UTC m=+1494.505734218"
Oct 30 14:13:49.779179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.779105    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:13:49.875649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.874876    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-kube-api-access-jhnz9\" (UniqueName: \"kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-kube-api-access-jhnz9\") pod \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") "
Oct 30 14:13:49.875649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.874942    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-envoy\") pod \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") "
Oct 30 14:13:49.875649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.875035    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-certs\") pod \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") "
Oct 30 14:13:49.875649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.875114    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istiod-ca-cert\") pod \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") "
Oct 30 14:13:49.875649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.875153    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-credential-socket\") pod \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") "
Oct 30 14:13:49.875649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.875181    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-token\" (UniqueName: \"kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-token\") pod \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") "
Oct 30 14:13:49.875649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.875373    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-socket\") pod \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") "
Oct 30 14:13:49.875649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.875443    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-data\") pod \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") "
Oct 30 14:13:49.875649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.875483    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-podinfo\") pod \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\" (UID: \"8e9979cc-ae59-4c43-abfe-88937c76fb2b\") "
Oct 30 14:13:49.876213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.875991    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istiod-ca-cert" pod "8e9979cc-ae59-4c43-abfe-88937c76fb2b" (UID: "8e9979cc-ae59-4c43-abfe-88937c76fb2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:49.876319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.876295    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:49.876644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.876567    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-credential-socket" pod "8e9979cc-ae59-4c43-abfe-88937c76fb2b" (UID: "8e9979cc-ae59-4c43-abfe-88937c76fb2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:49.876756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.876708    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-certs" pod "8e9979cc-ae59-4c43-abfe-88937c76fb2b" (UID: "8e9979cc-ae59-4c43-abfe-88937c76fb2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:49.876822 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.876782    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-data" pod "8e9979cc-ae59-4c43-abfe-88937c76fb2b" (UID: "8e9979cc-ae59-4c43-abfe-88937c76fb2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:49.877039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.877003    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-socket" pod "8e9979cc-ae59-4c43-abfe-88937c76fb2b" (UID: "8e9979cc-ae59-4c43-abfe-88937c76fb2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:49.889591 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.889084    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-podinfo" pod "8e9979cc-ae59-4c43-abfe-88937c76fb2b" (UID: "8e9979cc-ae59-4c43-abfe-88937c76fb2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:13:49.903776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.898643    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-envoy" pod "8e9979cc-ae59-4c43-abfe-88937c76fb2b" (UID: "8e9979cc-ae59-4c43-abfe-88937c76fb2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:49.905325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.904618    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-kube-api-access-jhnz9" pod "8e9979cc-ae59-4c43-abfe-88937c76fb2b" (UID: "8e9979cc-ae59-4c43-abfe-88937c76fb2b"). InnerVolumeSpecName "kube-api-access-jhnz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:49.905325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.904699    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-token" pod "8e9979cc-ae59-4c43-abfe-88937c76fb2b" (UID: "8e9979cc-ae59-4c43-abfe-88937c76fb2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:49.977943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.977904    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:49.977943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.977938    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:49.977943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.977948    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:49.978176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.977957    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:49.978176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.977965    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:49.978176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.977974    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:49.978176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.977982    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhnz9\" (UniqueName: \"kubernetes.io/projected/8e9979cc-ae59-4c43-abfe-88937c76fb2b-kube-api-access-jhnz9\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:49.978176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:49.977991    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e9979cc-ae59-4c43-abfe-88937c76fb2b-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:50.011566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.005634    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 14:13:50.011566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.005702    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 14:13:50.031016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.030944    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 14:13:50.031016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.030991    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 14:13:50.044613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.044579    4530 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 14:13:50.044720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.044629    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" podUID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 14:13:50.370766 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.369602    4530 generic.go:356] "Generic (PLEG): container finished" podID="025875c5-a966-4f05-bf7f-b68c9ba00b73" containerID="b6ae0d4bc6e27062b3f46b7c43e0b9c43c062a024bd51d4ae92fd7398323909e" exitCode=0
Oct 30 14:13:50.370766 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.369841    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls" event={"ID":"025875c5-a966-4f05-bf7f-b68c9ba00b73","Type":"ContainerDied","Data":"b6ae0d4bc6e27062b3f46b7c43e0b9c43c062a024bd51d4ae92fd7398323909e"}
Oct 30 14:13:50.382621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.373309    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q" event={"ID":"8e9979cc-ae59-4c43-abfe-88937c76fb2b","Type":"ContainerDied","Data":"34d349268c7991e783e4e6e182fbbc3045192134dcbcdae75604c620813e7e18"}
Oct 30 14:13:50.382621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.373378    4530 scope.go:122] "RemoveContainer" containerID="50af43fa9ee96aa5787ac51ef5cc82b3d02cbbec5776ca1eda233af45848b6a3"
Oct 30 14:13:50.382621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.373504    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"
Oct 30 14:13:50.512905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.512868    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"]
Oct 30 14:13:50.525735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:50.525704    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ea3e7b1d-openshift-default-65c65b785-8tt9q"]
Oct 30 14:13:51.178566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.169975    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" path="/var/lib/kubelet/pods/8e9979cc-ae59-4c43-abfe-88937c76fb2b/volumes"
Oct 30 14:13:51.452336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.447782    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:51.452336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.452105    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:51.457767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.457736    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:13:51.564255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.551581    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:51.564255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.552973    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:51.564255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.554918    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:13:51.702002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.701930    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:51.704092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.704066    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:51.704417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.704399    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:51.706275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.706255    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:13:51.813137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.813103    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-openshift-ca\") pod \"025875c5-a966-4f05-bf7f-b68c9ba00b73\" (UID: \"025875c5-a966-4f05-bf7f-b68c9ba00b73\") "
Oct 30 14:13:51.813332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.813204    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-xks-ca\") pod \"025875c5-a966-4f05-bf7f-b68c9ba00b73\" (UID: \"025875c5-a966-4f05-bf7f-b68c9ba00b73\") "
Oct 30 14:13:51.813332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.813317    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025875c5-a966-4f05-bf7f-b68c9ba00b73-kube-api-access-qv2c9\" (UniqueName: \"kubernetes.io/projected/025875c5-a966-4f05-bf7f-b68c9ba00b73-kube-api-access-qv2c9\") pod \"025875c5-a966-4f05-bf7f-b68c9ba00b73\" (UID: \"025875c5-a966-4f05-bf7f-b68c9ba00b73\") "
Oct 30 14:13:51.815598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.815454    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-openshift-ca" pod "025875c5-a966-4f05-bf7f-b68c9ba00b73" (UID: "025875c5-a966-4f05-bf7f-b68c9ba00b73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:51.818745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.818702    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025875c5-a966-4f05-bf7f-b68c9ba00b73-kube-api-access-qv2c9" pod "025875c5-a966-4f05-bf7f-b68c9ba00b73" (UID: "025875c5-a966-4f05-bf7f-b68c9ba00b73"). InnerVolumeSpecName "kube-api-access-qv2c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:51.833352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.833304    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-xks-ca" pod "025875c5-a966-4f05-bf7f-b68c9ba00b73" (UID: "025875c5-a966-4f05-bf7f-b68c9ba00b73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:51.916095 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.916050    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:51.916095 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.916096    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025875c5-a966-4f05-bf7f-b68c9ba00b73-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:51.916344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:51.916114    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv2c9\" (UniqueName: \"kubernetes.io/projected/025875c5-a966-4f05-bf7f-b68c9ba00b73-kube-api-access-qv2c9\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:13:52.469050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:52.468167    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"
Oct 30 14:13:52.469050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:52.468194    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls" event={"ID":"025875c5-a966-4f05-bf7f-b68c9ba00b73","Type":"ContainerDied","Data":"60a86378234ecdc06e04f2813e47a00dce4194762ac4d86e3d0f1a7e5aac22c3"}
Oct 30 14:13:52.469050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:52.468240    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60a86378234ecdc06e04f2813e47a00dce4194762ac4d86e3d0f1a7e5aac22c3"
Oct 30 14:13:54.582269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.582238    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"]
Oct 30 14:13:54.583007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.582990    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy"
Oct 30 14:13:54.583064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.583009    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy"
Oct 30 14:13:54.583064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.583031    4530 cpu_manager.go:414] "containerMap: removing container" podUID="025875c5-a966-4f05-bf7f-b68c9ba00b73" containerName="revoke-keys"
Oct 30 14:13:54.583064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.583038    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025875c5-a966-4f05-bf7f-b68c9ba00b73" containerName="revoke-keys"
Oct 30 14:13:54.583064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.583058    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy"
Oct 30 14:13:54.583064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.583063    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy"
Oct 30 14:13:54.583827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.583785    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy"
Oct 30 14:13:54.583941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.583857    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="025875c5-a966-4f05-bf7f-b68c9ba00b73" containerName="revoke-keys"
Oct 30 14:13:54.592432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.592402    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.607077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.607050    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"]
Oct 30 14:13:54.614437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.614409    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4522b19c-serving-cert\""
Oct 30 14:13:54.709817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.709779    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7shv7\" (UniqueName: \"kubernetes.io/projected/597ddf75-1053-462a-861e-f425bf22f4d3-kube-api-access-7shv7\") pod \"maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.709974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.709948    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-maas-api-tls\") pod \"maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.710066 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.710052    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-metrics-certs\") pod \"maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.710100 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.710080    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/597ddf75-1053-462a-861e-f425bf22f4d3-openshift-service-ca\") pod \"maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.811397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.811365    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-maas-api-tls\") pod \"maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.811561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.811481    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-metrics-certs\") pod \"maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.811561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.811506    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/597ddf75-1053-462a-861e-f425bf22f4d3-openshift-service-ca\") pod \"maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.811680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.811666    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7shv7\" (UniqueName: \"kubernetes.io/projected/597ddf75-1053-462a-861e-f425bf22f4d3-kube-api-access-7shv7\") pod \"maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.812293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.812272    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/597ddf75-1053-462a-861e-f425bf22f4d3-openshift-service-ca\") pod \"maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.814238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.814210    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-maas-api-tls\") pod \"maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.814457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.814441    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-metrics-certs\") pod \"maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.838449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.838391    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7shv7\" (UniqueName: \"kubernetes.io/projected/597ddf75-1053-462a-861e-f425bf22f4d3-kube-api-access-7shv7\") pod \"maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:54.914193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:54.914157    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:55.344213 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:13:55.344150    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597ddf75_1053_462a_861e_f425bf22f4d3.slice/crio-5c477553e8a125af3ad0ea6b764b5aaf777ae9e27199fbc8efef92045df5da83 WatchSource:0}: Error finding container 5c477553e8a125af3ad0ea6b764b5aaf777ae9e27199fbc8efef92045df5da83: Status 404 returned error can't find the container with id 5c477553e8a125af3ad0ea6b764b5aaf777ae9e27199fbc8efef92045df5da83
Oct 30 14:13:55.347664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:55.347641    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"]
Oct 30 14:13:55.491034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:55.490997    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:13:55.558215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:55.558180    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:13:55.561952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:55.561925    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:13:55.568042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:55.555189    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh" event={"ID":"597ddf75-1053-462a-861e-f425bf22f4d3","Type":"ContainerStarted","Data":"5c477553e8a125af3ad0ea6b764b5aaf777ae9e27199fbc8efef92045df5da83"}
Oct 30 14:13:55.607583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:55.607379    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:13:55.612438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:55.612415    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:13:55.613317 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:55.613299    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:13:56.565623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:56.565581    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh" event={"ID":"597ddf75-1053-462a-861e-f425bf22f4d3","Type":"ContainerStarted","Data":"afd3ee208fb8508eae37c89314e11bbbec3be995d385f611dc9622fd72b08fd0"}
Oct 30 14:13:56.565797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:56.565676    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:13:56.615207 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:56.615160    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh" podStartSLOduration=2.296123491 podStartE2EDuration="2.61514676s" totalImagesPullingTime="319.023269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:56.613201747 +0000 UTC m=+1501.645100687" watchObservedRunningTime="2026-10-30 14:13:56.61514676 +0000 UTC m=+1501.647045726"
Oct 30 14:13:58.962070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:58.955258    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"]
Oct 30 14:13:59.254524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:59.254199    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25917498-jb2wv"]
Oct 30 14:13:59.508617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:59.484553    4530 scope.go:122] "RemoveContainer" containerID="d0be799e94af7b70224c7b46d9baca182c3925ef14e181e2ac24e9153381c561"
Oct 30 14:13:59.676017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:59.675844    4530 scope.go:122] "RemoveContainer" containerID="86acecf15adbf4bc2b571582ea2c7118785f9e6d4a9c927e6634427ead70f450"
Oct 30 14:13:59.714435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:59.714408    4530 scope.go:122] "RemoveContainer" containerID="7344c0d086647a220b860108e1d489c16976ddfddd4472e259a2f696467a09e7"
Oct 30 14:13:59.731433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:13:59.731407    4530 scope.go:122] "RemoveContainer" containerID="d8b69b905690637ee40e567b3a757393b2cd4c88c4d49f2bd1945d35f64e555a"
Oct 30 14:14:01.049929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:01.049892    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"]
Oct 30 14:14:01.063400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:01.063367    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad9f7071-fn5z9"]
Oct 30 14:14:01.116226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:01.116198    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141128bf-6a3c-4894-b7d9-5502c5bcf339" path="/var/lib/kubelet/pods/141128bf-6a3c-4894-b7d9-5502c5bcf339/volumes"
Oct 30 14:14:01.116797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:01.116781    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e085efcd-2a7e-4798-adf0-fdb57b468308" path="/var/lib/kubelet/pods/e085efcd-2a7e-4798-adf0-fdb57b468308/volumes"
Oct 30 14:14:02.579824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:02.579795    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:14:04.690296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.690262    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"]
Oct 30 14:14:04.691100 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.691084    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9979cc-ae59-4c43-abfe-88937c76fb2b" containerName="istio-proxy"
Oct 30 14:14:04.697268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.697248    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:04.719041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.719009    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"]
Oct 30 14:14:04.722912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.722884    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-75443e-serving-cert\""
Oct 30 14:14:04.853672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.853640    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:04.853863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.853690    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d94sf\" (UniqueName: \"kubernetes.io/projected/781c45e8-e9fc-4928-b487-3371e529e361-kube-api-access-d94sf\") pod \"maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:04.853863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.853718    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:04.853995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.853874    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/781c45e8-e9fc-4928-b487-3371e529e361-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:04.963286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.963182    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94sf\" (UniqueName: \"kubernetes.io/projected/781c45e8-e9fc-4928-b487-3371e529e361-kube-api-access-d94sf\") pod \"maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:04.963996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.963955    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:04.965006 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.964982    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/781c45e8-e9fc-4928-b487-3371e529e361-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:04.965509 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.965486    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:04.979232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:04.979192    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/781c45e8-e9fc-4928-b487-3371e529e361-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:05.011584 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:05.006237    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94sf\" (UniqueName: \"kubernetes.io/projected/781c45e8-e9fc-4928-b487-3371e529e361-kube-api-access-d94sf\") pod \"maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:05.011819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:05.009395    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:05.044574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:05.043565    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:05.313040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:05.313011    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:05.740844 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:05.740809    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"]
Oct 30 14:14:05.764399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:05.764365    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq" event={"ID":"781c45e8-e9fc-4928-b487-3371e529e361","Type":"ContainerStarted","Data":"b619b6ad7456e6145e8b792c7947f84570e62037ba18378f25fc966f42662fa7"}
Oct 30 14:14:06.776769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:06.776728    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq" event={"ID":"781c45e8-e9fc-4928-b487-3371e529e361","Type":"ContainerStarted","Data":"b829244fc49b35da6a256ad2a0d98aca6699e7eb200be2df695abb47425461e9"}
Oct 30 14:14:06.777188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:06.776840    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:06.831961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:06.831916    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq" podStartSLOduration=2.500935574 podStartE2EDuration="2.831902725s" totalImagesPullingTime="330.967151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:06.825717557 +0000 UTC m=+1511.857616500" watchObservedRunningTime="2026-10-30 14:14:06.831902725 +0000 UTC m=+1511.863801706"
Oct 30 14:14:11.368690 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:14:11.368654    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:14:12.049512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:12.049471    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"]
Oct 30 14:14:12.062946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:12.062915    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1d377d-jgv8p"]
Oct 30 14:14:12.791151 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:12.791112    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:14:13.116766 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:13.116732    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64431e73-3911-42d5-b91e-89930c21b8aa" path="/var/lib/kubelet/pods/64431e73-3911-42d5-b91e-89930c21b8aa/volumes"
Oct 30 14:14:16.995560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:16.992279    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:14:22.066034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:22.065983    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"]
Oct 30 14:14:22.076346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:22.076314    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ff8aa5-4xwww"]
Oct 30 14:14:23.139578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:23.139514    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d26f449-ff5d-4ba6-8269-4e16d2f2d41a" path="/var/lib/kubelet/pods/1d26f449-ff5d-4ba6-8269-4e16d2f2d41a/volumes"
Oct 30 14:14:30.191364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:30.191329    4530 generic.go:356] "Generic (PLEG): container finished" podID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerID="b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167" exitCode=0
Oct 30 14:14:30.191870 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:30.191411    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" event={"ID":"b7bf62ef-287a-4ede-95e9-a0c8d8354141","Type":"ContainerDied","Data":"b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167"}
Oct 30 14:14:30.191870 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:30.191807    4530 scope.go:122] "RemoveContainer" containerID="b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167"
Oct 30 14:14:30.193512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:30.193478    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:14:30.193642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:30.193617    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:14:30.193724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:30.193700    4530 generic.go:356] "Generic (PLEG): container finished" podID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerID="6fbbe14c01ca93a04cc52a549e99536db241ebf2ff9299ea4ba9b4e92379482b" exitCode=0
Oct 30 14:14:30.193724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:30.193707    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:14:30.193841 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:30.193778    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" event={"ID":"9ee32a07-96da-46e2-86cc-b2deed6224fb","Type":"ContainerDied","Data":"6fbbe14c01ca93a04cc52a549e99536db241ebf2ff9299ea4ba9b4e92379482b"}
Oct 30 14:14:30.194139 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:30.194119    4530 scope.go:122] "RemoveContainer" containerID="6fbbe14c01ca93a04cc52a549e99536db241ebf2ff9299ea4ba9b4e92379482b"
Oct 30 14:14:30.195439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:30.195413    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:14:30.195502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:30.195474    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:14:30.195555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:30.195506    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:14:31.946622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:31.946187    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" event={"ID":"9ee32a07-96da-46e2-86cc-b2deed6224fb","Type":"ContainerStarted","Data":"316184b7e64d0817649f85b5959b0ce22e806afcef254a577aeec64864a59c21"}
Oct 30 14:14:31.949450 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:31.948907    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" event={"ID":"b7bf62ef-287a-4ede-95e9-a0c8d8354141","Type":"ContainerStarted","Data":"cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc"}
Oct 30 14:14:31.955907 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:31.955882    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:14:31.956914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:31.956890    4530 patch_prober.go:29] interesting pod/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 14:14:31.957014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:31.956933    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 14:14:32.136700 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:32.136661    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:14:32.137013 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:32.136712    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 14:14:32.137230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:32.137204    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 14:14:32.957892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:32.957858    4530 patch_prober.go:29] interesting pod/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 14:14:32.958576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:32.958527    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 14:14:33.128743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:33.128705    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 14:14:33.129052 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:33.129020    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 14:14:33.956096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:33.956048    4530 patch_prober.go:29] interesting pod/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 14:14:33.956437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:33.956387    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 14:14:34.103191 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:34.103153    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 14:14:34.103848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:34.103816    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 14:14:34.460756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:34.460686    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:14:34.729242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:34.729131    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:14:34.955252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:34.955215    4530 patch_prober.go:29] interesting pod/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 14:14:34.971901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:34.967634    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 14:14:35.109912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:35.100909    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 14:14:35.109912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:35.100984    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 14:14:35.730532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:35.730496    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:14:35.955267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:35.955218    4530 patch_prober.go:29] interesting pod/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 14:14:35.955438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:35.955282    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 14:14:36.106148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:36.106113    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 14:14:36.136570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:36.132321    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 14:14:36.955757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:36.955714    4530 patch_prober.go:29] interesting pod/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 14:14:36.955942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:36.955768    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 14:14:37.123578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:37.121571    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 14:14:37.123578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:37.121630    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 14:14:37.862439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:37.862407    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"]
Oct 30 14:14:37.862883 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:37.862617    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh" podUID="597ddf75-1053-462a-861e-f425bf22f4d3" containerName="maas-api" containerID="cri-o://afd3ee208fb8508eae37c89314e11bbbec3be995d385f611dc9622fd72b08fd0" gracePeriod=30
Oct 30 14:14:37.956176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:37.956142    4530 patch_prober.go:29] interesting pod/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 14:14:37.956360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:37.956278    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 14:14:38.265158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.265068    4530 generic.go:356] "Generic (PLEG): container finished" podID="597ddf75-1053-462a-861e-f425bf22f4d3" containerID="afd3ee208fb8508eae37c89314e11bbbec3be995d385f611dc9622fd72b08fd0" exitCode=0
Oct 30 14:14:38.265592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.265566    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh" event={"ID":"597ddf75-1053-462a-861e-f425bf22f4d3","Type":"ContainerDied","Data":"afd3ee208fb8508eae37c89314e11bbbec3be995d385f611dc9622fd72b08fd0"}
Oct 30 14:14:38.367394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.367362    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:14:38.542123 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.542031    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:14:38.562723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.562680    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:14:38.570741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.570707    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/597ddf75-1053-462a-861e-f425bf22f4d3-kube-api-access-7shv7\" (UniqueName: \"kubernetes.io/projected/597ddf75-1053-462a-861e-f425bf22f4d3-kube-api-access-7shv7\") pod \"597ddf75-1053-462a-861e-f425bf22f4d3\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") "
Oct 30 14:14:38.571514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.571487    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-metrics-certs\") pod \"597ddf75-1053-462a-861e-f425bf22f4d3\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") "
Oct 30 14:14:38.571887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.571822    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/597ddf75-1053-462a-861e-f425bf22f4d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/597ddf75-1053-462a-861e-f425bf22f4d3-openshift-service-ca\") pod \"597ddf75-1053-462a-861e-f425bf22f4d3\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") "
Oct 30 14:14:38.572101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.572082    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-maas-api-tls\") pod \"597ddf75-1053-462a-861e-f425bf22f4d3\" (UID: \"597ddf75-1053-462a-861e-f425bf22f4d3\") "
Oct 30 14:14:38.575592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.575569    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/597ddf75-1053-462a-861e-f425bf22f4d3-openshift-service-ca" pod "597ddf75-1053-462a-861e-f425bf22f4d3" (UID: "597ddf75-1053-462a-861e-f425bf22f4d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:14:38.744050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.666500    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-maas-api-tls" pod "597ddf75-1053-462a-861e-f425bf22f4d3" (UID: "597ddf75-1053-462a-861e-f425bf22f4d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:14:38.744050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.686418    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/597ddf75-1053-462a-861e-f425bf22f4d3-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:38.744050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.686453    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:38.744050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.687651    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597ddf75-1053-462a-861e-f425bf22f4d3-kube-api-access-7shv7" pod "597ddf75-1053-462a-861e-f425bf22f4d3" (UID: "597ddf75-1053-462a-861e-f425bf22f4d3"). InnerVolumeSpecName "kube-api-access-7shv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:14:38.744806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.744741    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-metrics-certs" pod "597ddf75-1053-462a-861e-f425bf22f4d3" (UID: "597ddf75-1053-462a-861e-f425bf22f4d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:14:38.806625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.803645    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/597ddf75-1053-462a-861e-f425bf22f4d3-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:38.806625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:38.803678    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7shv7\" (UniqueName: \"kubernetes.io/projected/597ddf75-1053-462a-861e-f425bf22f4d3-kube-api-access-7shv7\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:39.278423 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:39.278389    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"
Oct 30 14:14:39.279412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:39.279385    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh" event={"ID":"597ddf75-1053-462a-861e-f425bf22f4d3","Type":"ContainerDied","Data":"5c477553e8a125af3ad0ea6b764b5aaf777ae9e27199fbc8efef92045df5da83"}
Oct 30 14:14:39.279495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:39.279480    4530 scope.go:122] "RemoveContainer" containerID="afd3ee208fb8508eae37c89314e11bbbec3be995d385f611dc9622fd72b08fd0"
Oct 30 14:14:39.347630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:39.347600    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"]
Oct 30 14:14:39.354698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:39.354665    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4522b19c-6b5f4d598-9m7xh"]
Oct 30 14:14:39.377937 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:39.377905    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:14:39.379631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:39.379606    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:14:40.823664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:40.823626    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"]
Oct 30 14:14:41.182320 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:41.178848    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="597ddf75-1053-462a-861e-f425bf22f4d3" path="/var/lib/kubelet/pods/597ddf75-1053-462a-861e-f425bf22f4d3/volumes"
Oct 30 14:14:41.352008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:41.351958    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy" containerID="cri-o://316184b7e64d0817649f85b5959b0ce22e806afcef254a577aeec64864a59c21" gracePeriod=30
Oct 30 14:14:41.508417 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:14:41.508309    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:14:42.121402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.121364    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"]
Oct 30 14:14:42.122976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.122948    4530 cpu_manager.go:414] "containerMap: removing container" podUID="597ddf75-1053-462a-861e-f425bf22f4d3" containerName="maas-api"
Oct 30 14:14:42.123133 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.123120    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="597ddf75-1053-462a-861e-f425bf22f4d3" containerName="maas-api"
Oct 30 14:14:42.123426 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.123413    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="597ddf75-1053-462a-861e-f425bf22f4d3" containerName="maas-api"
Oct 30 14:14:42.127992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.127951    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.136653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.136620    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-data\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.136779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.136752    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-socket\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.136842 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.136801    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istiod-ca-cert\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.136842 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.136827    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-envoy\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.137286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.136860    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-podinfo\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.137286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.136957    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-token\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.137286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.137012    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-certs\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.137286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.137145    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-credential-socket\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.137286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.137177    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbxq7\" (UniqueName: \"kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-kube-api-access-xbxq7\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.155259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.155230    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"]
Oct 30 14:14:42.160680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.160648    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-e371c3-openshift-default-dockercfg-lb6vc\""
Oct 30 14:14:42.262572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.238753    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-credential-socket\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.313575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.295631    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbxq7\" (UniqueName: \"kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-kube-api-access-xbxq7\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.313575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.295852    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-data\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.313575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.296002    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-socket\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.313575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.296026    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istiod-ca-cert\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.313575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.296052    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-envoy\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.313575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.296082    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-podinfo\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.313575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.296186    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-token\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.313575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.296245    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-certs\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.313575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.312878    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-certs\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.313575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.313153    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-socket\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.313575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.264162    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-credential-socket\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.314806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.313780    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-data\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.316434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.316036    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istiod-ca-cert\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.346192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.346157    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-envoy\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.347690 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.347632    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-podinfo\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.452152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.452068    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-token\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.459399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.453835    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbxq7\" (UniqueName: \"kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-kube-api-access-xbxq7\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:42.745007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:42.744927    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:43.486261 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:14:43.484740    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaef6f1c_c355_4357_aa4c_f94ba2619dd9.slice/crio-19e29dfcb19d83d6a7821d6d26398d3b400643eb8513e18dc7ca448d6a022059 WatchSource:0}: Error finding container 19e29dfcb19d83d6a7821d6d26398d3b400643eb8513e18dc7ca448d6a022059: Status 404 returned error can't find the container with id 19e29dfcb19d83d6a7821d6d26398d3b400643eb8513e18dc7ca448d6a022059
Oct 30 14:14:43.533909 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:43.533874    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"]
Oct 30 14:14:43.534169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:43.534126    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:14:43.534340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:43.534308    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:14:43.534495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:43.534474    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:14:44.568592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:44.566052    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" event={"ID":"daef6f1c-c355-4357-aa4c-f94ba2619dd9","Type":"ContainerStarted","Data":"049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3"}
Oct 30 14:14:44.568592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:44.566106    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" event={"ID":"daef6f1c-c355-4357-aa4c-f94ba2619dd9","Type":"ContainerStarted","Data":"19e29dfcb19d83d6a7821d6d26398d3b400643eb8513e18dc7ca448d6a022059"}
Oct 30 14:14:44.669603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:44.667198    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" podStartSLOduration=2.667183286 podStartE2EDuration="2.667183286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:44.651123902 +0000 UTC m=+1549.683022843" watchObservedRunningTime="2026-10-30 14:14:44.667183286 +0000 UTC m=+1549.699082224"
Oct 30 14:14:44.748983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:44.748907    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:44.847621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:44.831491    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:45.138625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:45.118585    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"]
Oct 30 14:14:45.172565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:45.172280    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"]
Oct 30 14:14:45.198807 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:14:45.198756    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-f065f6cd-59qhv\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="7a1df436-5c89-4c81-b77a-61580da3b564" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f065f6cd-59qhv"
Oct 30 14:14:45.249051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:45.247610    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a1df436-5c89-4c81-b77a-61580da3b564" path="/var/lib/kubelet/pods/7a1df436-5c89-4c81-b77a-61580da3b564/volumes"
Oct 30 14:14:45.635566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:45.634736    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:45.654564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:45.643793    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:14:46.667623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:46.667281    4530 generic.go:356] "Generic (PLEG): container finished" podID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerID="316184b7e64d0817649f85b5959b0ce22e806afcef254a577aeec64864a59c21" exitCode=0
Oct 30 14:14:46.668173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:46.667653    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" event={"ID":"9ee32a07-96da-46e2-86cc-b2deed6224fb","Type":"ContainerDied","Data":"316184b7e64d0817649f85b5959b0ce22e806afcef254a577aeec64864a59c21"}
Oct 30 14:14:46.668173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:46.667760    4530 scope.go:122] "RemoveContainer" containerID="6fbbe14c01ca93a04cc52a549e99536db241ebf2ff9299ea4ba9b4e92379482b"
Oct 30 14:14:46.961260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:46.961234    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:14:47.037789 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.037727    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-certs\") pod \"9ee32a07-96da-46e2-86cc-b2deed6224fb\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") "
Oct 30 14:14:47.038790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.038756    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-certs" pod "9ee32a07-96da-46e2-86cc-b2deed6224fb" (UID: "9ee32a07-96da-46e2-86cc-b2deed6224fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:14:47.038936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.038842    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-data\") pod \"9ee32a07-96da-46e2-86cc-b2deed6224fb\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") "
Oct 30 14:14:47.038936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.038893    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-token\" (UniqueName: \"kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-token\") pod \"9ee32a07-96da-46e2-86cc-b2deed6224fb\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") "
Oct 30 14:14:47.039050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.038932    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-podinfo\") pod \"9ee32a07-96da-46e2-86cc-b2deed6224fb\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") "
Oct 30 14:14:47.039108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.039052    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-credential-socket\") pod \"9ee32a07-96da-46e2-86cc-b2deed6224fb\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") "
Oct 30 14:14:47.039218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.039202    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-kube-api-access-xqv8c\" (UniqueName: \"kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-kube-api-access-xqv8c\") pod \"9ee32a07-96da-46e2-86cc-b2deed6224fb\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") "
Oct 30 14:14:47.039298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.039272    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-data" pod "9ee32a07-96da-46e2-86cc-b2deed6224fb" (UID: "9ee32a07-96da-46e2-86cc-b2deed6224fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:14:47.039354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.039280    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-socket\") pod \"9ee32a07-96da-46e2-86cc-b2deed6224fb\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") "
Oct 30 14:14:47.039354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.039343    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee32a07-96da-46e2-86cc-b2deed6224fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee32a07-96da-46e2-86cc-b2deed6224fb-istiod-ca-cert\") pod \"9ee32a07-96da-46e2-86cc-b2deed6224fb\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") "
Oct 30 14:14:47.039463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.039447    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-envoy\") pod \"9ee32a07-96da-46e2-86cc-b2deed6224fb\" (UID: \"9ee32a07-96da-46e2-86cc-b2deed6224fb\") "
Oct 30 14:14:47.040297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.040270    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:47.040401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.040300    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:47.040678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.040642    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-credential-socket" pod "9ee32a07-96da-46e2-86cc-b2deed6224fb" (UID: "9ee32a07-96da-46e2-86cc-b2deed6224fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:14:47.041604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.041102    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-socket" pod "9ee32a07-96da-46e2-86cc-b2deed6224fb" (UID: "9ee32a07-96da-46e2-86cc-b2deed6224fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:14:47.041604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.041494    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee32a07-96da-46e2-86cc-b2deed6224fb-istiod-ca-cert" pod "9ee32a07-96da-46e2-86cc-b2deed6224fb" (UID: "9ee32a07-96da-46e2-86cc-b2deed6224fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:14:47.041939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.041916    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-token" pod "9ee32a07-96da-46e2-86cc-b2deed6224fb" (UID: "9ee32a07-96da-46e2-86cc-b2deed6224fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:14:47.043426 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.043386    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-podinfo" pod "9ee32a07-96da-46e2-86cc-b2deed6224fb" (UID: "9ee32a07-96da-46e2-86cc-b2deed6224fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:14:47.043528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.043403    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-envoy" pod "9ee32a07-96da-46e2-86cc-b2deed6224fb" (UID: "9ee32a07-96da-46e2-86cc-b2deed6224fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:14:47.044153 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.044128    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-kube-api-access-xqv8c" pod "9ee32a07-96da-46e2-86cc-b2deed6224fb" (UID: "9ee32a07-96da-46e2-86cc-b2deed6224fb"). InnerVolumeSpecName "kube-api-access-xqv8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:14:47.142169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.142140    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqv8c\" (UniqueName: \"kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-kube-api-access-xqv8c\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:47.142169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.142167    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:47.142343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.142181    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee32a07-96da-46e2-86cc-b2deed6224fb-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:47.142343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.142193    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:47.142343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.142202    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:47.142343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.142209    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee32a07-96da-46e2-86cc-b2deed6224fb-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:47.142343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.142216    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee32a07-96da-46e2-86cc-b2deed6224fb-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:14:47.677883 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.677855    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"
Oct 30 14:14:47.678368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.677876    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464" event={"ID":"9ee32a07-96da-46e2-86cc-b2deed6224fb","Type":"ContainerDied","Data":"c45000f617d9e4ee2effd35f070fe24560049f598f3e0358e3bb2a7fb1532305"}
Oct 30 14:14:47.678368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.677954    4530 scope.go:122] "RemoveContainer" containerID="316184b7e64d0817649f85b5959b0ce22e806afcef254a577aeec64864a59c21"
Oct 30 14:14:47.748359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.748325    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"]
Oct 30 14:14:47.762096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:47.762071    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4522b19c-openshift-default-b9d85f877-wj464"]
Oct 30 14:14:49.429132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.429099    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" path="/var/lib/kubelet/pods/9ee32a07-96da-46e2-86cc-b2deed6224fb/volumes"
Oct 30 14:14:49.430244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.430180    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"]
Oct 30 14:14:49.431897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.431876    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy"
Oct 30 14:14:49.432023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.432010    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy"
Oct 30 14:14:49.432341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.432325    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy"
Oct 30 14:14:49.442430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.442398    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"]
Oct 30 14:14:49.442775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.442690    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.461904 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.461453    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9dba4af9-serving-cert\""
Oct 30 14:14:49.472456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.472398    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-metrics-certs\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.472610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.472522    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-maas-api-tls\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.472728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.472704    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c41d290-df32-406e-badd-688c5e316f53-openshift-service-ca\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.472843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.472824    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rdhc\" (UniqueName: \"kubernetes.io/projected/1c41d290-df32-406e-badd-688c5e316f53-kube-api-access-7rdhc\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.574263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.574233    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-metrics-certs\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.574389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.574290    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-maas-api-tls\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.574389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.574344    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c41d290-df32-406e-badd-688c5e316f53-openshift-service-ca\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.574503 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.574418    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rdhc\" (UniqueName: \"kubernetes.io/projected/1c41d290-df32-406e-badd-688c5e316f53-kube-api-access-7rdhc\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.575125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.575099    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c41d290-df32-406e-badd-688c5e316f53-openshift-service-ca\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.576744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.576722    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-maas-api-tls\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.576834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.576758    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-metrics-certs\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.686619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.681196    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rdhc\" (UniqueName: \"kubernetes.io/projected/1c41d290-df32-406e-badd-688c5e316f53-kube-api-access-7rdhc\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:49.778610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:49.770467    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:50.140558 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:14:50.140513    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c41d290_df32_406e_badd_688c5e316f53.slice/crio-18a8575de2fbabbdc0aa59110f1740542db00f0b623e389d17d547a601180edd WatchSource:0}: Error finding container 18a8575de2fbabbdc0aa59110f1740542db00f0b623e389d17d547a601180edd: Status 404 returned error can't find the container with id 18a8575de2fbabbdc0aa59110f1740542db00f0b623e389d17d547a601180edd
Oct 30 14:14:50.143933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:50.143909    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"]
Oct 30 14:14:50.753578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:50.753508    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf" event={"ID":"1c41d290-df32-406e-badd-688c5e316f53","Type":"ContainerStarted","Data":"def33dafede5ec5b8fc50d42b94d81072ac0280ec80cbc749eb176322dc0f332"}
Oct 30 14:14:50.753971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:50.753613    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:50.753971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:50.753640    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf" event={"ID":"1c41d290-df32-406e-badd-688c5e316f53","Type":"ContainerStarted","Data":"18a8575de2fbabbdc0aa59110f1740542db00f0b623e389d17d547a601180edd"}
Oct 30 14:14:50.850687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:50.845814    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf" podStartSLOduration=1.416860371 podStartE2EDuration="1.845790512s" totalImagesPullingTime="428.930141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:50.82426078 +0000 UTC m=+1555.856159722" watchObservedRunningTime="2026-10-30 14:14:50.845790512 +0000 UTC m=+1555.877689454"
Oct 30 14:14:56.061001 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:56.060963    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"]
Oct 30 14:14:56.074147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:56.074117    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557ad36c-sdc8r"]
Oct 30 14:14:56.768271 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:56.768241    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:14:57.116278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:57.116237    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9681345-6f5c-4527-9b68-e86a1f66b15d" path="/var/lib/kubelet/pods/a9681345-6f5c-4527-9b68-e86a1f66b15d/volumes"
Oct 30 14:14:57.703253 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:57.703218    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:14:59.845010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:59.844976    4530 scope.go:122] "RemoveContainer" containerID="db87d9600234a251759006bb730c1f9c116cf36bcca453849b4001c443851a68"
Oct 30 14:14:59.855704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:59.855678    4530 scope.go:122] "RemoveContainer" containerID="7ffb8e96fd80b82039b9a0d01e0cf9a31530dd38e2897e8dd55e4d8ff346f548"
Oct 30 14:14:59.865843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:14:59.865821    4530 scope.go:122] "RemoveContainer" containerID="ee6b3bc0d65bb574d390d06a34cc2f284e0a80600c37df6168cb714d9f96be0b"
Oct 30 14:15:00.180103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.179955    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"]
Oct 30 14:15:00.180787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.180767    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy"
Oct 30 14:15:00.180787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.180786    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy"
Oct 30 14:15:00.180968 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.180946    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee32a07-96da-46e2-86cc-b2deed6224fb" containerName="istio-proxy"
Oct 30 14:15:00.186761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.186741    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"]
Oct 30 14:15:00.186881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.186864    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"
Oct 30 14:15:00.192583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.192534    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"
Oct 30 14:15:00.222221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.222196    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"]
Oct 30 14:15:00.253642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.253608    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"]
Oct 30 14:15:00.259213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.259184    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"]
Oct 30 14:15:00.265206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.265180    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"
Oct 30 14:15:00.267168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.267143    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"]
Oct 30 14:15:00.273764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.273743    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"
Oct 30 14:15:00.285796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.285769    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"]
Oct 30 14:15:00.294900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.294879    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:15:00.297177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.297150    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"]
Oct 30 14:15:00.311664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.311642    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57\" (UID: \"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"
Oct 30 14:15:00.312105 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.312082    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdc6q\" (UniqueName: \"kubernetes.io/projected/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-kube-api-access-xdc6q\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57\" (UID: \"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"
Oct 30 14:15:00.312177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.312139    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94\" (UID: \"1a2926f5-ce9a-4cbb-9225-3651a7121d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"
Oct 30 14:15:00.312259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.312235    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cspm9\" (UniqueName: \"kubernetes.io/projected/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-kube-api-access-cspm9\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94\" (UID: \"1a2926f5-ce9a-4cbb-9225-3651a7121d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"
Oct 30 14:15:00.413236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.413206    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57\" (UID: \"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"
Oct 30 14:15:00.413335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.413259    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4nm\" (UniqueName: \"kubernetes.io/projected/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-kube-api-access-2m4nm\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv\" (UID: \"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"
Oct 30 14:15:00.413335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.413319    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69c1867-add2-41c7-a66f-fffd8caafe8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg\" (UID: \"f69c1867-add2-41c7-a66f-fffd8caafe8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"
Oct 30 14:15:00.413486 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.413463    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdc6q\" (UniqueName: \"kubernetes.io/projected/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-kube-api-access-xdc6q\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57\" (UID: \"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"
Oct 30 14:15:00.413588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.413515    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfvrr\" (UniqueName: \"kubernetes.io/projected/f69c1867-add2-41c7-a66f-fffd8caafe8d-kube-api-access-mfvrr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg\" (UID: \"f69c1867-add2-41c7-a66f-fffd8caafe8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"
Oct 30 14:15:00.413667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.413598    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94\" (UID: \"1a2926f5-ce9a-4cbb-9225-3651a7121d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"
Oct 30 14:15:00.413667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.413630    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cspm9\" (UniqueName: \"kubernetes.io/projected/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-kube-api-access-cspm9\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94\" (UID: \"1a2926f5-ce9a-4cbb-9225-3651a7121d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"
Oct 30 14:15:00.413774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.413674    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv\" (UID: \"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"
Oct 30 14:15:00.413873 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.413854    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57\" (UID: \"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"
Oct 30 14:15:00.414184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.414166    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94\" (UID: \"1a2926f5-ce9a-4cbb-9225-3651a7121d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"
Oct 30 14:15:00.440753 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.440700    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cspm9\" (UniqueName: \"kubernetes.io/projected/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-kube-api-access-cspm9\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94\" (UID: \"1a2926f5-ce9a-4cbb-9225-3651a7121d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"
Oct 30 14:15:00.440930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.440909    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdc6q\" (UniqueName: \"kubernetes.io/projected/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-kube-api-access-xdc6q\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57\" (UID: \"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"
Oct 30 14:15:00.515270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.515241    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4nm\" (UniqueName: \"kubernetes.io/projected/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-kube-api-access-2m4nm\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv\" (UID: \"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"
Oct 30 14:15:00.515378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.515345    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69c1867-add2-41c7-a66f-fffd8caafe8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg\" (UID: \"f69c1867-add2-41c7-a66f-fffd8caafe8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"
Oct 30 14:15:00.515477 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.515436    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfvrr\" (UniqueName: \"kubernetes.io/projected/f69c1867-add2-41c7-a66f-fffd8caafe8d-kube-api-access-mfvrr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg\" (UID: \"f69c1867-add2-41c7-a66f-fffd8caafe8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"
Oct 30 14:15:00.515559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.515480    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv\" (UID: \"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"
Oct 30 14:15:00.516019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.515987    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69c1867-add2-41c7-a66f-fffd8caafe8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg\" (UID: \"f69c1867-add2-41c7-a66f-fffd8caafe8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"
Oct 30 14:15:00.516196 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.516177    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv\" (UID: \"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"
Oct 30 14:15:00.520343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.520327    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"
Oct 30 14:15:00.530519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.530500    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"
Oct 30 14:15:00.570961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.570931    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4nm\" (UniqueName: \"kubernetes.io/projected/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-kube-api-access-2m4nm\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv\" (UID: \"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"
Oct 30 14:15:00.571267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.571244    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfvrr\" (UniqueName: \"kubernetes.io/projected/f69c1867-add2-41c7-a66f-fffd8caafe8d-kube-api-access-mfvrr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg\" (UID: \"f69c1867-add2-41c7-a66f-fffd8caafe8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"
Oct 30 14:15:00.589679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.589377    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"
Oct 30 14:15:00.598430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.598398    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"
Oct 30 14:15:00.842500 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:15:00.842468    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1612cd_a28e_4913_b2e0_b57a2b6dfcb7.slice/crio-d56b7020a74fbb526ed771492ffce2f04abed5121020d4fb0e889e02d2a433f3 WatchSource:0}: Error finding container d56b7020a74fbb526ed771492ffce2f04abed5121020d4fb0e889e02d2a433f3: Status 404 returned error can't find the container with id d56b7020a74fbb526ed771492ffce2f04abed5121020d4fb0e889e02d2a433f3
Oct 30 14:15:00.846412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.846386    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"]
Oct 30 14:15:00.857254 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:15:00.857223    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2926f5_ce9a_4cbb_9225_3651a7121d6e.slice/crio-8597df00096c6c042bd25b305104756d56dc50feb7abf4e773138786858b5942 WatchSource:0}: Error finding container 8597df00096c6c042bd25b305104756d56dc50feb7abf4e773138786858b5942: Status 404 returned error can't find the container with id 8597df00096c6c042bd25b305104756d56dc50feb7abf4e773138786858b5942
Oct 30 14:15:00.861682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.861659    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"]
Oct 30 14:15:00.888777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.888751    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57" event={"ID":"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7","Type":"ContainerStarted","Data":"d56b7020a74fbb526ed771492ffce2f04abed5121020d4fb0e889e02d2a433f3"}
Oct 30 14:15:00.890161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.890139    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94" event={"ID":"1a2926f5-ce9a-4cbb-9225-3651a7121d6e","Type":"ContainerStarted","Data":"8597df00096c6c042bd25b305104756d56dc50feb7abf4e773138786858b5942"}
Oct 30 14:15:00.924813 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:15:00.924781    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe7fc8f_eb9e_41ef_a530_21d96c50b16d.slice/crio-6cb9638fd8f49f7b99e9010a31fbc45ac10be35176f3c7c87c073119b4020f44 WatchSource:0}: Error finding container 6cb9638fd8f49f7b99e9010a31fbc45ac10be35176f3c7c87c073119b4020f44: Status 404 returned error can't find the container with id 6cb9638fd8f49f7b99e9010a31fbc45ac10be35176f3c7c87c073119b4020f44
Oct 30 14:15:00.926813 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:15:00.926792    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69c1867_add2_41c7_a66f_fffd8caafe8d.slice/crio-f5483255b2cdcca30f3c73619973ad127ecb5d3c994ea26b2c57b94a8a5bb138 WatchSource:0}: Error finding container f5483255b2cdcca30f3c73619973ad127ecb5d3c994ea26b2c57b94a8a5bb138: Status 404 returned error can't find the container with id f5483255b2cdcca30f3c73619973ad127ecb5d3c994ea26b2c57b94a8a5bb138
Oct 30 14:15:00.931391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.930424    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"]
Oct 30 14:15:00.937926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:00.937902    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"]
Oct 30 14:15:01.902446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:01.902416    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv" event={"ID":"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d","Type":"ContainerStarted","Data":"59ac4105072e6f35b8040ef5d9c3ce252e80aebdea4da7bbb17242776242fee6"}
Oct 30 14:15:01.902800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:01.902476    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv" event={"ID":"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d","Type":"ContainerStarted","Data":"6cb9638fd8f49f7b99e9010a31fbc45ac10be35176f3c7c87c073119b4020f44"}
Oct 30 14:15:01.904090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:01.904064    4530 generic.go:356] "Generic (PLEG): container finished" podID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" containerID="892c9635f4f4fab673d8b6ddba1a2d718af8f550fabf7ace73e8665770f75f95" exitCode=60
Oct 30 14:15:01.904177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:01.904133    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57" event={"ID":"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7","Type":"ContainerDied","Data":"892c9635f4f4fab673d8b6ddba1a2d718af8f550fabf7ace73e8665770f75f95"}
Oct 30 14:15:01.904385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:01.904368    4530 scope.go:122] "RemoveContainer" containerID="892c9635f4f4fab673d8b6ddba1a2d718af8f550fabf7ace73e8665770f75f95"
Oct 30 14:15:01.905936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:01.905915    4530 generic.go:356] "Generic (PLEG): container finished" podID="f69c1867-add2-41c7-a66f-fffd8caafe8d" containerID="fbcf0466cfeff22acae8ad9f31acbd097835e11a7b1b35b5b9bb43bd923724db" exitCode=60
Oct 30 14:15:01.906046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:01.905946    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg" event={"ID":"f69c1867-add2-41c7-a66f-fffd8caafe8d","Type":"ContainerDied","Data":"fbcf0466cfeff22acae8ad9f31acbd097835e11a7b1b35b5b9bb43bd923724db"}
Oct 30 14:15:01.906046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:01.905998    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg" event={"ID":"f69c1867-add2-41c7-a66f-fffd8caafe8d","Type":"ContainerStarted","Data":"f5483255b2cdcca30f3c73619973ad127ecb5d3c994ea26b2c57b94a8a5bb138"}
Oct 30 14:15:01.906261 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:01.906244    4530 scope.go:122] "RemoveContainer" containerID="fbcf0466cfeff22acae8ad9f31acbd097835e11a7b1b35b5b9bb43bd923724db"
Oct 30 14:15:01.908047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:01.908026    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94" event={"ID":"1a2926f5-ce9a-4cbb-9225-3651a7121d6e","Type":"ContainerStarted","Data":"3ec00b6758a7e30794f98e0d23a053e736a92628ba88ca602253fef6851695ef"}
Oct 30 14:15:01.949246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:01.949177    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv" podStartSLOduration=1.949161613 podStartE2EDuration="1.949161613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:01.94669432 +0000 UTC m=+1566.978593261" watchObservedRunningTime="2026-10-30 14:15:01.949161613 +0000 UTC m=+1566.981060553"
Oct 30 14:15:02.041193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.041148    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94" podStartSLOduration=2.041135658 podStartE2EDuration="2.041135658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:02.03750202 +0000 UTC m=+1567.069400958" watchObservedRunningTime="2026-10-30 14:15:02.041135658 +0000 UTC m=+1567.073034596"
Oct 30 14:15:02.918249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.918219    4530 generic.go:356] "Generic (PLEG): container finished" podID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" containerID="59ac4105072e6f35b8040ef5d9c3ce252e80aebdea4da7bbb17242776242fee6" exitCode=60
Oct 30 14:15:02.918626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.918298    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv" event={"ID":"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d","Type":"ContainerDied","Data":"59ac4105072e6f35b8040ef5d9c3ce252e80aebdea4da7bbb17242776242fee6"}
Oct 30 14:15:02.918626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.918590    4530 scope.go:122] "RemoveContainer" containerID="59ac4105072e6f35b8040ef5d9c3ce252e80aebdea4da7bbb17242776242fee6"
Oct 30 14:15:02.920499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.920479    4530 generic.go:356] "Generic (PLEG): container finished" podID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" containerID="90928c998b1be682c20c50a036f2fff25f409a02bc6a5a689d8961f30e48bd4a" exitCode=60
Oct 30 14:15:02.920628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.920572    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57" event={"ID":"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7","Type":"ContainerDied","Data":"90928c998b1be682c20c50a036f2fff25f409a02bc6a5a689d8961f30e48bd4a"}
Oct 30 14:15:02.920702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.920641    4530 scope.go:122] "RemoveContainer" containerID="892c9635f4f4fab673d8b6ddba1a2d718af8f550fabf7ace73e8665770f75f95"
Oct 30 14:15:02.920921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.920904    4530 scope.go:122] "RemoveContainer" containerID="90928c998b1be682c20c50a036f2fff25f409a02bc6a5a689d8961f30e48bd4a"
Oct 30 14:15:02.921251 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:02.921225    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57_odh-ai-gateway-infra(2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7"
Oct 30 14:15:02.922893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.922874    4530 generic.go:356] "Generic (PLEG): container finished" podID="f69c1867-add2-41c7-a66f-fffd8caafe8d" containerID="62c9cc22c05a948335f977a2ac8fa3308b95f8e26d39e474c53f1daceea1a37d" exitCode=60
Oct 30 14:15:02.922970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.922953    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg" event={"ID":"f69c1867-add2-41c7-a66f-fffd8caafe8d","Type":"ContainerDied","Data":"62c9cc22c05a948335f977a2ac8fa3308b95f8e26d39e474c53f1daceea1a37d"}
Oct 30 14:15:02.923237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.923219    4530 scope.go:122] "RemoveContainer" containerID="62c9cc22c05a948335f977a2ac8fa3308b95f8e26d39e474c53f1daceea1a37d"
Oct 30 14:15:02.923522 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:02.923500    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg_odh-ai-gateway-infra(f69c1867-add2-41c7-a66f-fffd8caafe8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d"
Oct 30 14:15:02.930465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.930443    4530 generic.go:356] "Generic (PLEG): container finished" podID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" containerID="3ec00b6758a7e30794f98e0d23a053e736a92628ba88ca602253fef6851695ef" exitCode=60
Oct 30 14:15:02.930593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.930522    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94" event={"ID":"1a2926f5-ce9a-4cbb-9225-3651a7121d6e","Type":"ContainerDied","Data":"3ec00b6758a7e30794f98e0d23a053e736a92628ba88ca602253fef6851695ef"}
Oct 30 14:15:02.930759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.930743    4530 scope.go:122] "RemoveContainer" containerID="3ec00b6758a7e30794f98e0d23a053e736a92628ba88ca602253fef6851695ef"
Oct 30 14:15:02.950784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:02.950761    4530 scope.go:122] "RemoveContainer" containerID="fbcf0466cfeff22acae8ad9f31acbd097835e11a7b1b35b5b9bb43bd923724db"
Oct 30 14:15:03.945765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:03.945734    4530 generic.go:356] "Generic (PLEG): container finished" podID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" containerID="37ced1eac8ddf4ec23af62e18aeee855dd4da2b821a91563183e791fa4d27de4" exitCode=60
Oct 30 14:15:03.946178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:03.946119    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv" event={"ID":"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d","Type":"ContainerDied","Data":"37ced1eac8ddf4ec23af62e18aeee855dd4da2b821a91563183e791fa4d27de4"}
Oct 30 14:15:03.946251 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:03.946225    4530 scope.go:122] "RemoveContainer" containerID="59ac4105072e6f35b8040ef5d9c3ce252e80aebdea4da7bbb17242776242fee6"
Oct 30 14:15:03.946319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:03.946248    4530 scope.go:122] "RemoveContainer" containerID="37ced1eac8ddf4ec23af62e18aeee855dd4da2b821a91563183e791fa4d27de4"
Oct 30 14:15:03.947042 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:03.946716    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv_odh-ai-gateway-infra(1fe7fc8f-eb9e-41ef-a530-21d96c50b16d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d"
Oct 30 14:15:03.949507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:03.949477    4530 scope.go:122] "RemoveContainer" containerID="90928c998b1be682c20c50a036f2fff25f409a02bc6a5a689d8961f30e48bd4a"
Oct 30 14:15:03.949879 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:03.949851    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57_odh-ai-gateway-infra(2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7"
Oct 30 14:15:03.951175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:03.951159    4530 scope.go:122] "RemoveContainer" containerID="62c9cc22c05a948335f977a2ac8fa3308b95f8e26d39e474c53f1daceea1a37d"
Oct 30 14:15:03.951473 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:03.951452    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg_odh-ai-gateway-infra(f69c1867-add2-41c7-a66f-fffd8caafe8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d"
Oct 30 14:15:03.952931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:03.952910    4530 generic.go:356] "Generic (PLEG): container finished" podID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" containerID="e35c1d9b41ded23196e1e5d00675e8bcce7f6e4507f869d2003768fa60dad4b8" exitCode=60
Oct 30 14:15:03.953024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:03.952994    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94" event={"ID":"1a2926f5-ce9a-4cbb-9225-3651a7121d6e","Type":"ContainerDied","Data":"e35c1d9b41ded23196e1e5d00675e8bcce7f6e4507f869d2003768fa60dad4b8"}
Oct 30 14:15:03.953175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:03.953161    4530 scope.go:122] "RemoveContainer" containerID="e35c1d9b41ded23196e1e5d00675e8bcce7f6e4507f869d2003768fa60dad4b8"
Oct 30 14:15:03.953427 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:03.953411    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94_odh-ai-gateway-infra(1a2926f5-ce9a-4cbb-9225-3651a7121d6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e"
Oct 30 14:15:03.980587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:03.980533    4530 scope.go:122] "RemoveContainer" containerID="3ec00b6758a7e30794f98e0d23a053e736a92628ba88ca602253fef6851695ef"
Oct 30 14:15:04.963495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:04.963465    4530 scope.go:122] "RemoveContainer" containerID="37ced1eac8ddf4ec23af62e18aeee855dd4da2b821a91563183e791fa4d27de4"
Oct 30 14:15:04.963914 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:04.963842    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv_odh-ai-gateway-infra(1fe7fc8f-eb9e-41ef-a530-21d96c50b16d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d"
Oct 30 14:15:04.965224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:04.965206    4530 scope.go:122] "RemoveContainer" containerID="e35c1d9b41ded23196e1e5d00675e8bcce7f6e4507f869d2003768fa60dad4b8"
Oct 30 14:15:04.965523 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:04.965503    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94_odh-ai-gateway-infra(1a2926f5-ce9a-4cbb-9225-3651a7121d6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e"
Oct 30 14:15:09.044943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.044855    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"]
Oct 30 14:15:09.049816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.049796    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.072505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.072479    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"]
Oct 30 14:15:09.077707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.077682    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-e371c3-serving-cert\""
Oct 30 14:15:09.197086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.197047    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.197266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.197130    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.197323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.197255    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v679w\" (UniqueName: \"kubernetes.io/projected/3214e243-6b64-417d-8711-4b3cacb3c1bb-kube-api-access-v679w\") pod \"maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.197570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.197516    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3214e243-6b64-417d-8711-4b3cacb3c1bb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.309618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.307563    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.309618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.307641    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v679w\" (UniqueName: \"kubernetes.io/projected/3214e243-6b64-417d-8711-4b3cacb3c1bb-kube-api-access-v679w\") pod \"maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.309618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.307935    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3214e243-6b64-417d-8711-4b3cacb3c1bb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.309618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.308039    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.318562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.316914    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3214e243-6b64-417d-8711-4b3cacb3c1bb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.333711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.331559    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.333711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.333520    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.379623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.374434    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v679w\" (UniqueName: \"kubernetes.io/projected/3214e243-6b64-417d-8711-4b3cacb3c1bb-kube-api-access-v679w\") pod \"maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.666479 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.666401    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:09.977796 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:15:09.977755    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3214e243_6b64_417d_8711_4b3cacb3c1bb.slice/crio-47efef1e93c0bb4cc142b1368797c3c58cc518facee1db189c5b104a4fa00a30 WatchSource:0}: Error finding container 47efef1e93c0bb4cc142b1368797c3c58cc518facee1db189c5b104a4fa00a30: Status 404 returned error can't find the container with id 47efef1e93c0bb4cc142b1368797c3c58cc518facee1db189c5b104a4fa00a30
Oct 30 14:15:09.981631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:09.981604    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"]
Oct 30 14:15:10.010148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:10.010115    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w" event={"ID":"3214e243-6b64-417d-8711-4b3cacb3c1bb","Type":"ContainerStarted","Data":"47efef1e93c0bb4cc142b1368797c3c58cc518facee1db189c5b104a4fa00a30"}
Oct 30 14:15:11.020911 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:11.020868    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w" event={"ID":"3214e243-6b64-417d-8711-4b3cacb3c1bb","Type":"ContainerStarted","Data":"35ad71c79a7ea1ac1085520dba1645871f8da5a06524074c3aed2e76b38a690a"}
Oct 30 14:15:11.021292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:11.020958    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:11.071105 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:11.071062    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w" podStartSLOduration=1.726356793 podStartE2EDuration="2.071050372s" totalImagesPullingTime="344.693579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:11.067639148 +0000 UTC m=+1576.099538087" watchObservedRunningTime="2026-10-30 14:15:11.071050372 +0000 UTC m=+1576.102949311"
Oct 30 14:15:11.403592 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:11.403521    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:15:15.162586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:15.160302    4530 scope.go:122] "RemoveContainer" containerID="90928c998b1be682c20c50a036f2fff25f409a02bc6a5a689d8961f30e48bd4a"
Oct 30 14:15:16.048729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:16.048694    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"]
Oct 30 14:15:16.062997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:16.062966    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48391366-4znkv"]
Oct 30 14:15:16.108087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:16.108065    4530 scope.go:122] "RemoveContainer" containerID="62c9cc22c05a948335f977a2ac8fa3308b95f8e26d39e474c53f1daceea1a37d"
Oct 30 14:15:16.108639 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:16.108617    4530 scope.go:122] "RemoveContainer" containerID="e35c1d9b41ded23196e1e5d00675e8bcce7f6e4507f869d2003768fa60dad4b8"
Oct 30 14:15:16.125311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:16.125288    4530 generic.go:356] "Generic (PLEG): container finished" podID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" containerID="27cabc6ec2e3e51bfed042d9c58a1902071602f8593edbf5f6956f88f7ac430c" exitCode=60
Oct 30 14:15:16.125426 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:16.125365    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57" event={"ID":"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7","Type":"ContainerDied","Data":"27cabc6ec2e3e51bfed042d9c58a1902071602f8593edbf5f6956f88f7ac430c"}
Oct 30 14:15:16.125487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:16.125427    4530 scope.go:122] "RemoveContainer" containerID="90928c998b1be682c20c50a036f2fff25f409a02bc6a5a689d8961f30e48bd4a"
Oct 30 14:15:16.125782 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:16.125760    4530 scope.go:122] "RemoveContainer" containerID="27cabc6ec2e3e51bfed042d9c58a1902071602f8593edbf5f6956f88f7ac430c"
Oct 30 14:15:16.126074 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:16.126052    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57_odh-ai-gateway-infra(2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7"
Oct 30 14:15:17.061876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.061851    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:15:17.107622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.107597    4530 scope.go:122] "RemoveContainer" containerID="37ced1eac8ddf4ec23af62e18aeee855dd4da2b821a91563183e791fa4d27de4"
Oct 30 14:15:17.120365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.119956    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac9d860c-4982-42b2-bea5-418b31649631" path="/var/lib/kubelet/pods/ac9d860c-4982-42b2-bea5-418b31649631/volumes"
Oct 30 14:15:17.167892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.166481    4530 generic.go:356] "Generic (PLEG): container finished" podID="f69c1867-add2-41c7-a66f-fffd8caafe8d" containerID="d64d0fdfc634ba9412d80d682c028111f1dab4ae5b97b32a78b6caedfd3bc1a7" exitCode=60
Oct 30 14:15:17.167892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.166643    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg" event={"ID":"f69c1867-add2-41c7-a66f-fffd8caafe8d","Type":"ContainerDied","Data":"d64d0fdfc634ba9412d80d682c028111f1dab4ae5b97b32a78b6caedfd3bc1a7"}
Oct 30 14:15:17.167892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.166705    4530 scope.go:122] "RemoveContainer" containerID="62c9cc22c05a948335f977a2ac8fa3308b95f8e26d39e474c53f1daceea1a37d"
Oct 30 14:15:17.167892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.167186    4530 scope.go:122] "RemoveContainer" containerID="d64d0fdfc634ba9412d80d682c028111f1dab4ae5b97b32a78b6caedfd3bc1a7"
Oct 30 14:15:17.167892 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:17.167566    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg_odh-ai-gateway-infra(f69c1867-add2-41c7-a66f-fffd8caafe8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d"
Oct 30 14:15:17.170764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.170740    4530 generic.go:356] "Generic (PLEG): container finished" podID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" containerID="e71fdea7492a005b7027908b16a118b6f7035071bff139f79fbcaf2427fa46e4" exitCode=60
Oct 30 14:15:17.170894 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.170812    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94" event={"ID":"1a2926f5-ce9a-4cbb-9225-3651a7121d6e","Type":"ContainerDied","Data":"e71fdea7492a005b7027908b16a118b6f7035071bff139f79fbcaf2427fa46e4"}
Oct 30 14:15:17.171251 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.171223    4530 scope.go:122] "RemoveContainer" containerID="e71fdea7492a005b7027908b16a118b6f7035071bff139f79fbcaf2427fa46e4"
Oct 30 14:15:17.171584 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:17.171561    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94_odh-ai-gateway-infra(1a2926f5-ce9a-4cbb-9225-3651a7121d6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e"
Oct 30 14:15:17.187578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.187554    4530 scope.go:122] "RemoveContainer" containerID="e35c1d9b41ded23196e1e5d00675e8bcce7f6e4507f869d2003768fa60dad4b8"
Oct 30 14:15:17.214260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.214194    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"]
Oct 30 14:15:17.436744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.436720    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"
Oct 30 14:15:17.542640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.533700    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-maas-api-service-ca\") pod \"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7\" (UID: \"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7\") "
Oct 30 14:15:17.542640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.533975    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-kube-api-access-xdc6q\" (UniqueName: \"kubernetes.io/projected/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-kube-api-access-xdc6q\") pod \"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7\" (UID: \"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7\") "
Oct 30 14:15:17.542640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.540681    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-maas-api-service-ca" pod "2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" (UID: "2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:15:17.587746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.587171    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-kube-api-access-xdc6q" pod "2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" (UID: "2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7"). InnerVolumeSpecName "kube-api-access-xdc6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:15:17.639530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.637107    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:15:17.639530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:17.637141    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdc6q\" (UniqueName: \"kubernetes.io/projected/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7-kube-api-access-xdc6q\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:15:18.182145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.182118    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"
Oct 30 14:15:18.182145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.182127    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57" event={"ID":"2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7","Type":"ContainerDied","Data":"d56b7020a74fbb526ed771492ffce2f04abed5121020d4fb0e889e02d2a433f3"}
Oct 30 14:15:18.182712 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.182184    4530 scope.go:122] "RemoveContainer" containerID="27cabc6ec2e3e51bfed042d9c58a1902071602f8593edbf5f6956f88f7ac430c"
Oct 30 14:15:18.188351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.188321    4530 generic.go:356] "Generic (PLEG): container finished" podID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" containerID="c270501dea2368f457dc2a06d7b288a422299718600961ba74cbdf7acefaab2c" exitCode=60
Oct 30 14:15:18.188476 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.188375    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv" event={"ID":"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d","Type":"ContainerDied","Data":"c270501dea2368f457dc2a06d7b288a422299718600961ba74cbdf7acefaab2c"}
Oct 30 14:15:18.188968 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.188831    4530 scope.go:122] "RemoveContainer" containerID="c270501dea2368f457dc2a06d7b288a422299718600961ba74cbdf7acefaab2c"
Oct 30 14:15:18.189186 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:18.189160    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv_odh-ai-gateway-infra(1fe7fc8f-eb9e-41ef-a530-21d96c50b16d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d"
Oct 30 14:15:18.197798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.197780    4530 scope.go:122] "RemoveContainer" containerID="37ced1eac8ddf4ec23af62e18aeee855dd4da2b821a91563183e791fa4d27de4"
Oct 30 14:15:18.232813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.232792    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"]
Oct 30 14:15:18.302664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.294977    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"]
Oct 30 14:15:18.303663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.303632    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889495-v8h57"]
Oct 30 14:15:18.352230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.352196    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"]
Oct 30 14:15:18.424907 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.424868    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"
Oct 30 14:15:18.529783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.529760    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"
Oct 30 14:15:18.552842 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.552818    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69c1867-add2-41c7-a66f-fffd8caafe8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69c1867-add2-41c7-a66f-fffd8caafe8d-maas-api-service-ca\") pod \"f69c1867-add2-41c7-a66f-fffd8caafe8d\" (UID: \"f69c1867-add2-41c7-a66f-fffd8caafe8d\") "
Oct 30 14:15:18.553002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.552990    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69c1867-add2-41c7-a66f-fffd8caafe8d-kube-api-access-mfvrr\" (UniqueName: \"kubernetes.io/projected/f69c1867-add2-41c7-a66f-fffd8caafe8d-kube-api-access-mfvrr\") pod \"f69c1867-add2-41c7-a66f-fffd8caafe8d\" (UID: \"f69c1867-add2-41c7-a66f-fffd8caafe8d\") "
Oct 30 14:15:18.553200 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.553175    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69c1867-add2-41c7-a66f-fffd8caafe8d-maas-api-service-ca" pod "f69c1867-add2-41c7-a66f-fffd8caafe8d" (UID: "f69c1867-add2-41c7-a66f-fffd8caafe8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:15:18.553589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.553530    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69c1867-add2-41c7-a66f-fffd8caafe8d-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:15:18.555039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.555021    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69c1867-add2-41c7-a66f-fffd8caafe8d-kube-api-access-mfvrr" pod "f69c1867-add2-41c7-a66f-fffd8caafe8d" (UID: "f69c1867-add2-41c7-a66f-fffd8caafe8d"). InnerVolumeSpecName "kube-api-access-mfvrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:15:18.655025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.654991    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-maas-api-service-ca\") pod \"1a2926f5-ce9a-4cbb-9225-3651a7121d6e\" (UID: \"1a2926f5-ce9a-4cbb-9225-3651a7121d6e\") "
Oct 30 14:15:18.655272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.655250    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-kube-api-access-cspm9\" (UniqueName: \"kubernetes.io/projected/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-kube-api-access-cspm9\") pod \"1a2926f5-ce9a-4cbb-9225-3651a7121d6e\" (UID: \"1a2926f5-ce9a-4cbb-9225-3651a7121d6e\") "
Oct 30 14:15:18.655416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.655389    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-maas-api-service-ca" pod "1a2926f5-ce9a-4cbb-9225-3651a7121d6e" (UID: "1a2926f5-ce9a-4cbb-9225-3651a7121d6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:15:18.656047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.656027    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:15:18.656144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.656050    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfvrr\" (UniqueName: \"kubernetes.io/projected/f69c1867-add2-41c7-a66f-fffd8caafe8d-kube-api-access-mfvrr\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:15:18.657978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.657947    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-kube-api-access-cspm9" pod "1a2926f5-ce9a-4cbb-9225-3651a7121d6e" (UID: "1a2926f5-ce9a-4cbb-9225-3651a7121d6e"). InnerVolumeSpecName "kube-api-access-cspm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:15:18.758020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:18.757992    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cspm9\" (UniqueName: \"kubernetes.io/projected/1a2926f5-ce9a-4cbb-9225-3651a7121d6e-kube-api-access-cspm9\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:15:19.117247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.117210    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" path="/var/lib/kubelet/pods/2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7/volumes"
Oct 30 14:15:19.200681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.200660    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"
Oct 30 14:15:19.200681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.200672    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg" event={"ID":"f69c1867-add2-41c7-a66f-fffd8caafe8d","Type":"ContainerDied","Data":"f5483255b2cdcca30f3c73619973ad127ecb5d3c994ea26b2c57b94a8a5bb138"}
Oct 30 14:15:19.201122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.200735    4530 scope.go:122] "RemoveContainer" containerID="d64d0fdfc634ba9412d80d682c028111f1dab4ae5b97b32a78b6caedfd3bc1a7"
Oct 30 14:15:19.202350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.202324    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"
Oct 30 14:15:19.202463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.202358    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94" event={"ID":"1a2926f5-ce9a-4cbb-9225-3651a7121d6e","Type":"ContainerDied","Data":"8597df00096c6c042bd25b305104756d56dc50feb7abf4e773138786858b5942"}
Oct 30 14:15:19.211900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.211882    4530 scope.go:122] "RemoveContainer" containerID="e71fdea7492a005b7027908b16a118b6f7035071bff139f79fbcaf2427fa46e4"
Oct 30 14:15:19.274160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.274077    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"]
Oct 30 14:15:19.281773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.281742    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"]
Oct 30 14:15:19.288201 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.288169    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889495-2wjcg"]
Oct 30 14:15:19.310656 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.310625    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"]
Oct 30 14:15:19.316062 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.316035    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889495-gnl94"]
Oct 30 14:15:19.584324 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.584299    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"
Oct 30 14:15:19.645163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.645131    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-kube-api-access-2m4nm\" (UniqueName: \"kubernetes.io/projected/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-kube-api-access-2m4nm\") pod \"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d\" (UID: \"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d\") "
Oct 30 14:15:19.645353 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.645185    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-maas-api-service-ca\") pod \"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d\" (UID: \"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d\") "
Oct 30 14:15:19.645785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.645734    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-maas-api-service-ca" pod "1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" (UID: "1fe7fc8f-eb9e-41ef-a530-21d96c50b16d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:15:19.648119 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.648089    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-kube-api-access-2m4nm" pod "1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" (UID: "1fe7fc8f-eb9e-41ef-a530-21d96c50b16d"). InnerVolumeSpecName "kube-api-access-2m4nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:15:19.747386 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.747355    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m4nm\" (UniqueName: \"kubernetes.io/projected/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-kube-api-access-2m4nm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:15:19.747386 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:19.747383    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:15:20.212156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:20.212122    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"
Oct 30 14:15:20.212605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:20.212158    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv" event={"ID":"1fe7fc8f-eb9e-41ef-a530-21d96c50b16d","Type":"ContainerDied","Data":"6cb9638fd8f49f7b99e9010a31fbc45ac10be35176f3c7c87c073119b4020f44"}
Oct 30 14:15:20.212605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:20.212233    4530 scope.go:122] "RemoveContainer" containerID="c270501dea2368f457dc2a06d7b288a422299718600961ba74cbdf7acefaab2c"
Oct 30 14:15:20.279274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:20.279249    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"]
Oct 30 14:15:20.284797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:20.284770    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889495-kvkwv"]
Oct 30 14:15:21.116902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:21.116861    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" path="/var/lib/kubelet/pods/1a2926f5-ce9a-4cbb-9225-3651a7121d6e/volumes"
Oct 30 14:15:21.117342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:21.117328    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" path="/var/lib/kubelet/pods/1fe7fc8f-eb9e-41ef-a530-21d96c50b16d/volumes"
Oct 30 14:15:21.117774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:21.117760    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d" path="/var/lib/kubelet/pods/f69c1867-add2-41c7-a66f-fffd8caafe8d/volumes"
Oct 30 14:15:27.459565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.458604    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"]
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.459852    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.459874    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.459894    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.459903    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.459941    4530 cpu_manager.go:414] "containerMap: removing container" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.459949    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.459959    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.459966    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.459990    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.459997    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460010    4530 cpu_manager.go:414] "containerMap: removing container" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460017    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460054    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460061    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460071    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" containerName="cleanup"
Oct 30 14:15:27.460070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460078    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" containerName="cleanup"
Oct 30 14:15:27.460761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460248    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" containerName="cleanup"
Oct 30 14:15:27.460761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460260    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" containerName="cleanup"
Oct 30 14:15:27.460761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460274    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" containerName="cleanup"
Oct 30 14:15:27.460761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460281    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d" containerName="cleanup"
Oct 30 14:15:27.460761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460291    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d" containerName="cleanup"
Oct 30 14:15:27.460761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460298    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" containerName="cleanup"
Oct 30 14:15:27.460761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460312    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" containerName="cleanup"
Oct 30 14:15:27.460761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460323    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" containerName="cleanup"
Oct 30 14:15:27.460761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.460330    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" containerName="cleanup"
Oct 30 14:15:27.688587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.687696    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.691491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.691460    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"]
Oct 30 14:15:27.710285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.710200    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-5704d6-openshift-default-dockercfg-fhvn8\""
Oct 30 14:15:27.827562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.815748    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac20408-a58d-4bc9-8526-b86fe83d9020-istiod-ca-cert\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.827562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.815823    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-token\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.827562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.815891    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-socket\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.827562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.816009    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrzdn\" (UniqueName: \"kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-kube-api-access-jrzdn\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.827562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.816077    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-data\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.827562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.816136    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-certs\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.827562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.816246    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-podinfo\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.827562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.816324    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-envoy\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.827562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.816379    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-credential-socket\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.918327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.918290    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-token\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.918622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.918605    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-socket\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.918823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.918809    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrzdn\" (UniqueName: \"kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-kube-api-access-jrzdn\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.918976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.918964    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-data\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.919098 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.919084    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-certs\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.919267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.919252    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-podinfo\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.919412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.919400    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-envoy\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.919530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.919518    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-credential-socket\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.919685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.919671    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac20408-a58d-4bc9-8526-b86fe83d9020-istiod-ca-cert\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.920884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.920853    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac20408-a58d-4bc9-8526-b86fe83d9020-istiod-ca-cert\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.921637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.921616    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-socket\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.922216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.922195    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-data\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.922577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.922560    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-certs\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.926413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.926387    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-credential-socket\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.927157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.927115    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-podinfo\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.929460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.929440    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-envoy\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:27.991187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.991110    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-token\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:28.001375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:27.999342    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrzdn\" (UniqueName: \"kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-kube-api-access-jrzdn\") pod \"e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") " pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:28.069319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:28.069261    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"]
Oct 30 14:15:28.075600 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:28.075574    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4414981d-h9ngx"]
Oct 30 14:15:28.234106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:28.234067    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:28.569090 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:15:28.569059    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac20408_a58d_4bc9_8526_b86fe83d9020.slice/crio-97c8b36ca8f9aac798cd8ff90b145f3098e84d25a91edec2565135bfa455c968 WatchSource:0}: Error finding container 97c8b36ca8f9aac798cd8ff90b145f3098e84d25a91edec2565135bfa455c968: Status 404 returned error can't find the container with id 97c8b36ca8f9aac798cd8ff90b145f3098e84d25a91edec2565135bfa455c968
Oct 30 14:15:28.571665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:28.571628    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:15:28.571744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:28.571699    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:15:28.571744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:28.571735    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:15:28.572775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:28.572754    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"]
Oct 30 14:15:29.169568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:29.167240    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8672c30c-ad28-4aff-bda5-1adbcf8dd456" path="/var/lib/kubelet/pods/8672c30c-ad28-4aff-bda5-1adbcf8dd456/volumes"
Oct 30 14:15:29.470361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:29.470275    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" event={"ID":"7ac20408-a58d-4bc9-8526-b86fe83d9020","Type":"ContainerStarted","Data":"3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0"}
Oct 30 14:15:29.470361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:29.470329    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" event={"ID":"7ac20408-a58d-4bc9-8526-b86fe83d9020","Type":"ContainerStarted","Data":"97c8b36ca8f9aac798cd8ff90b145f3098e84d25a91edec2565135bfa455c968"}
Oct 30 14:15:29.666431 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:29.666395    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:15:29.692563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:29.689800    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" podStartSLOduration=2.689779599 podStartE2EDuration="2.689779599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:29.600128659 +0000 UTC m=+1594.632027623" watchObservedRunningTime="2026-10-30 14:15:29.689779599 +0000 UTC m=+1594.721678538"
Oct 30 14:15:30.235211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:30.235178    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:30.240677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:30.240652    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:30.479296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:30.479262    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:30.480219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:30.480196    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:15:38.836264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.836227    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"]
Oct 30 14:15:38.837466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.837441    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d" containerName="cleanup"
Oct 30 14:15:38.837657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.837469    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d" containerName="cleanup"
Oct 30 14:15:38.837657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.837488    4530 cpu_manager.go:414] "containerMap: removing container" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" containerName="cleanup"
Oct 30 14:15:38.837657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.837497    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1612cd-a28e-4913-b2e0-b57a2b6dfcb7" containerName="cleanup"
Oct 30 14:15:38.837657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.837579    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" containerName="cleanup"
Oct 30 14:15:38.837657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.837588    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" containerName="cleanup"
Oct 30 14:15:38.837657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.837603    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" containerName="cleanup"
Oct 30 14:15:38.837657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.837611    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" containerName="cleanup"
Oct 30 14:15:38.838053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.837794    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69c1867-add2-41c7-a66f-fffd8caafe8d" containerName="cleanup"
Oct 30 14:15:38.838053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.837821    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2926f5-ce9a-4cbb-9225-3651a7121d6e" containerName="cleanup"
Oct 30 14:15:38.838053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.837832    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe7fc8f-eb9e-41ef-a530-21d96c50b16d" containerName="cleanup"
Oct 30 14:15:38.843981 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.843957    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:38.872498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.872448    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a59ecb-openshift-default-dockercfg-q4p9h\""
Oct 30 14:15:38.873429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.873399    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-data\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:38.873598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.873536    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd9b019-951d-47cf-a812-946f4e10c8ba-istiod-ca-cert\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:38.879865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.879836    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-certs\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:38.880005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.879952    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-credential-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:38.880072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.880008    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-envoy\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:38.880072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.880060    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-token\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:38.880190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.880121    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc7q9\" (UniqueName: \"kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-kube-api-access-bc7q9\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:38.880190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.880167    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-podinfo\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:38.880302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.880245    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:38.883761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:38.883734    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"]
Oct 30 14:15:39.057565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.032999    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-token\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.071709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.071660    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc7q9\" (UniqueName: \"kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-kube-api-access-bc7q9\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.158956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.158861    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-podinfo\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.159323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.159297    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.159519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.159492    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-data\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.159881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.159858    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd9b019-951d-47cf-a812-946f4e10c8ba-istiod-ca-cert\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.162154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.162130    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-certs\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.162416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.162397    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-credential-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.189566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.163124    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd9b019-951d-47cf-a812-946f4e10c8ba-istiod-ca-cert\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.233589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.161082    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc7q9\" (UniqueName: \"kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-kube-api-access-bc7q9\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.300619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.174607    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-envoy\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.300619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.161574    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.300619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.175487    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-credential-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.300619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.162480    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-data\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.335111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.335074    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-token\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.369874 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.367187    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9603-243730"]
Oct 30 14:15:39.517880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.517785    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9603-243730"]
Oct 30 14:15:39.518586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.518560    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-243730"
Oct 30 14:15:39.529772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.529736    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-envoy\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.590803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.590766    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-certs\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.596188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.592856    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q95l7\""
Oct 30 14:15:39.623735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.623514    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-podinfo\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.633748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.633711    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8jph\" (UniqueName: \"kubernetes.io/projected/e8530935-762e-43b0-bf08-3dd3149686be-kube-api-access-b8jph\") pod \"test-curl-9603-243730\" (UID: \"e8530935-762e-43b0-bf08-3dd3149686be\") " pod="opendatahub/test-curl-9603-243730"
Oct 30 14:15:39.736472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.736066    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8jph\" (UniqueName: \"kubernetes.io/projected/e8530935-762e-43b0-bf08-3dd3149686be-kube-api-access-b8jph\") pod \"test-curl-9603-243730\" (UID: \"e8530935-762e-43b0-bf08-3dd3149686be\") " pod="opendatahub/test-curl-9603-243730"
Oct 30 14:15:39.763903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.763868    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8jph\" (UniqueName: \"kubernetes.io/projected/e8530935-762e-43b0-bf08-3dd3149686be-kube-api-access-b8jph\") pod \"test-curl-9603-243730\" (UID: \"e8530935-762e-43b0-bf08-3dd3149686be\") " pod="opendatahub/test-curl-9603-243730"
Oct 30 14:15:39.788277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.788201    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:39.898711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:39.898674    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-243730"
Oct 30 14:15:40.113984 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:15:40.113947    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd9b019_951d_47cf_a812_946f4e10c8ba.slice/crio-3d99e83e0b751cac9bba4c41abe76d5e06bda8bbfd35c0cc46fbe0f99faad02c WatchSource:0}: Error finding container 3d99e83e0b751cac9bba4c41abe76d5e06bda8bbfd35c0cc46fbe0f99faad02c: Status 404 returned error can't find the container with id 3d99e83e0b751cac9bba4c41abe76d5e06bda8bbfd35c0cc46fbe0f99faad02c
Oct 30 14:15:40.117446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:40.117413    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:15:40.117579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:40.117483    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:15:40.117579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:40.117511    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:15:40.117914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:40.117884    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"]
Oct 30 14:15:40.214203 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:15:40.214170    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8530935_762e_43b0_bf08_3dd3149686be.slice/crio-c06fe39ca29fe79924704b8314b100c34eef8389c5d124d6df6a02471e09c345 WatchSource:0}: Error finding container c06fe39ca29fe79924704b8314b100c34eef8389c5d124d6df6a02471e09c345: Status 404 returned error can't find the container with id c06fe39ca29fe79924704b8314b100c34eef8389c5d124d6df6a02471e09c345
Oct 30 14:15:40.219104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:40.219068    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9603-243730"]
Oct 30 14:15:40.878470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:40.878434    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" event={"ID":"7bd9b019-951d-47cf-a812-946f4e10c8ba","Type":"ContainerStarted","Data":"eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4"}
Oct 30 14:15:40.878683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:40.878489    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" event={"ID":"7bd9b019-951d-47cf-a812-946f4e10c8ba","Type":"ContainerStarted","Data":"3d99e83e0b751cac9bba4c41abe76d5e06bda8bbfd35c0cc46fbe0f99faad02c"}
Oct 30 14:15:40.880206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:40.880175    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9603-243730" event={"ID":"e8530935-762e-43b0-bf08-3dd3149686be","Type":"ContainerStarted","Data":"3d50cdd2e8f9a757410d488a614f6a7dab1fc2f432b7ec2b8e824d3109e15cb0"}
Oct 30 14:15:40.880326 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:40.880217    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9603-243730" event={"ID":"e8530935-762e-43b0-bf08-3dd3149686be","Type":"ContainerStarted","Data":"c06fe39ca29fe79924704b8314b100c34eef8389c5d124d6df6a02471e09c345"}
Oct 30 14:15:40.946792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:40.946728    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" podStartSLOduration=2.946711675 podStartE2EDuration="2.946711675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:40.944768664 +0000 UTC m=+1605.976667600" watchObservedRunningTime="2026-10-30 14:15:40.946711675 +0000 UTC m=+1605.978610615"
Oct 30 14:15:40.995923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:40.995871    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9603-243730" podStartSLOduration=1.995856759 podStartE2EDuration="1.995856759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:40.990121572 +0000 UTC m=+1606.022020511" watchObservedRunningTime="2026-10-30 14:15:40.995856759 +0000 UTC m=+1606.027755698"
Oct 30 14:15:41.043659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:41.043624    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"]
Oct 30 14:15:41.049388 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:41.049360    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5beda1-wthzr"]
Oct 30 14:15:41.201714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:41.198271    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4079be-b302-497b-9e34-abe0e7208507" path="/var/lib/kubelet/pods/ec4079be-b302-497b-9e34-abe0e7208507/volumes"
Oct 30 14:15:41.369693 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:41.369649    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:15:41.789192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:41.789157    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:41.794755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:41.794731    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:41.903392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:41.903358    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:41.904424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:41.904401    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:15:41.913236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:41.913205    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9603-243730"]
Oct 30 14:15:41.920785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:41.920755    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9603-243730"]
Oct 30 14:15:42.083364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.083332    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9603-568a02"]
Oct 30 14:15:42.084286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.084262    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e8530935-762e-43b0-bf08-3dd3149686be" containerName="test-curl-9603-243730"
Oct 30 14:15:42.084422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.084288    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8530935-762e-43b0-bf08-3dd3149686be" containerName="test-curl-9603-243730"
Oct 30 14:15:42.084587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.084564    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8530935-762e-43b0-bf08-3dd3149686be" containerName="test-curl-9603-243730"
Oct 30 14:15:42.090221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.090196    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-568a02"
Oct 30 14:15:42.106003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.105971    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9603-568a02"]
Oct 30 14:15:42.162371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.162313    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"]
Oct 30 14:15:42.168111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.168088    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.186380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.186165    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"]
Oct 30 14:15:42.189581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.189556    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9dba4af9\"/\"default-dockercfg-6mmzg\""
Oct 30 14:15:42.189581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.189577    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9dba4af9\"/\"openshift-service-ca.crt\""
Oct 30 14:15:42.189805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.189642    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9dba4af9\"/\"e2e-lifecycle-model-9dba4af9-kserve-self-signed-certs\""
Oct 30 14:15:42.189805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.189718    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9dba4af9\"/\"kube-root-ca.crt\""
Oct 30 14:15:42.234729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.234694    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cpt2\" (UniqueName: \"kubernetes.io/projected/e51dd13e-af5e-49d9-a333-414e5f493923-kube-api-access-7cpt2\") pod \"test-curl-9603-568a02\" (UID: \"e51dd13e-af5e-49d9-a333-414e5f493923\") " pod="opendatahub/test-curl-9603-568a02"
Oct 30 14:15:42.234976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.234860    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-dshm\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.234976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.234910    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tmp-dir\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.235245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.235220    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tls-certs\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.235317 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.235281    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-home\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.235420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.235401    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7w7\" (UniqueName: \"kubernetes.io/projected/91d82203-dd8e-4e1f-ad76-0403aea6eaca-kube-api-access-zr7w7\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.235482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.235469    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-model-cache\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.344625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.340078    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cpt2\" (UniqueName: \"kubernetes.io/projected/e51dd13e-af5e-49d9-a333-414e5f493923-kube-api-access-7cpt2\") pod \"test-curl-9603-568a02\" (UID: \"e51dd13e-af5e-49d9-a333-414e5f493923\") " pod="opendatahub/test-curl-9603-568a02"
Oct 30 14:15:42.344625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.340144    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-dshm\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.344625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.340192    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tmp-dir\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.344625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.340399    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tls-certs\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.344625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.340449    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-home\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.344625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.340501    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7w7\" (UniqueName: \"kubernetes.io/projected/91d82203-dd8e-4e1f-ad76-0403aea6eaca-kube-api-access-zr7w7\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.344625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.340555    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-model-cache\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.344625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.341244    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-model-cache\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.347868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.346829    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-home\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.347868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.347159    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tmp-dir\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.400562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.363848    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-dshm\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.400562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.380258    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tls-certs\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.409173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.409105    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7w7\" (UniqueName: \"kubernetes.io/projected/91d82203-dd8e-4e1f-ad76-0403aea6eaca-kube-api-access-zr7w7\") pod \"e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") " pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.434580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.426315    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cpt2\" (UniqueName: \"kubernetes.io/projected/e51dd13e-af5e-49d9-a333-414e5f493923-kube-api-access-7cpt2\") pod \"test-curl-9603-568a02\" (UID: \"e51dd13e-af5e-49d9-a333-414e5f493923\") " pod="opendatahub/test-curl-9603-568a02"
Oct 30 14:15:42.486893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.486854    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:42.712864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.712788    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-568a02"
Oct 30 14:15:42.913481 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.913424    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9603-243730" podUID="e8530935-762e-43b0-bf08-3dd3149686be" containerName="test-curl-9603-243730" containerID="cri-o://3d50cdd2e8f9a757410d488a614f6a7dab1fc2f432b7ec2b8e824d3109e15cb0" gracePeriod=2
Oct 30 14:15:42.928998 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:42.928961    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-243730\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e8530935-762e-43b0-bf08-3dd3149686be" pod="opendatahub/test-curl-9603-243730"
Oct 30 14:15:42.986343 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:15:42.986305    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91d82203_dd8e_4e1f_ad76_0403aea6eaca.slice/crio-36411decd111e87cbda651610384474b606011e932e572bbd713cfa7bd847d64 WatchSource:0}: Error finding container 36411decd111e87cbda651610384474b606011e932e572bbd713cfa7bd847d64: Status 404 returned error can't find the container with id 36411decd111e87cbda651610384474b606011e932e572bbd713cfa7bd847d64
Oct 30 14:15:42.989311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:42.989286    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"]
Oct 30 14:15:43.063512 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:15:43.063478    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode51dd13e_af5e_49d9_a333_414e5f493923.slice/crio-7b382a03735dc932163c857052273820e4600fa22e0ad3d941ebaf47f219fbfa WatchSource:0}: Error finding container 7b382a03735dc932163c857052273820e4600fa22e0ad3d941ebaf47f219fbfa: Status 404 returned error can't find the container with id 7b382a03735dc932163c857052273820e4600fa22e0ad3d941ebaf47f219fbfa
Oct 30 14:15:43.066666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:43.066644    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9603-568a02"]
Oct 30 14:15:44.010326 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.010287    4530 generic.go:356] "Generic (PLEG): container finished" podID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerID="049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3" exitCode=0
Oct 30 14:15:44.011181 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.011150    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" event={"ID":"daef6f1c-c355-4357-aa4c-f94ba2619dd9","Type":"ContainerDied","Data":"049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3"}
Oct 30 14:15:44.012351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.012138    4530 scope.go:122] "RemoveContainer" containerID="049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3"
Oct 30 14:15:44.014815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.014782    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:15:44.014917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.014875    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:15:44.014982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.014923    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:15:44.017176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.017082    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr" event={"ID":"91d82203-dd8e-4e1f-ad76-0403aea6eaca","Type":"ContainerStarted","Data":"3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4"}
Oct 30 14:15:44.017176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.017124    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr" event={"ID":"91d82203-dd8e-4e1f-ad76-0403aea6eaca","Type":"ContainerStarted","Data":"36411decd111e87cbda651610384474b606011e932e572bbd713cfa7bd847d64"}
Oct 30 14:15:44.021872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.021477    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9603-568a02" event={"ID":"e51dd13e-af5e-49d9-a333-414e5f493923","Type":"ContainerStarted","Data":"85574c8bef999895e7935a21b2b6d9cbf6264ce53672a0a3fac185c28ef4ab1b"}
Oct 30 14:15:44.021872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.021512    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9603-568a02" event={"ID":"e51dd13e-af5e-49d9-a333-414e5f493923","Type":"ContainerStarted","Data":"7b382a03735dc932163c857052273820e4600fa22e0ad3d941ebaf47f219fbfa"}
Oct 30 14:15:44.118033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.117979    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9603-568a02" podStartSLOduration=2.117958616 podStartE2EDuration="2.117958616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:44.113392242 +0000 UTC m=+1609.145291184" watchObservedRunningTime="2026-10-30 14:15:44.117958616 +0000 UTC m=+1609.149857556"
Oct 30 14:15:44.176011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.175945    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr" podStartSLOduration=2.175926207 podStartE2EDuration="2.175926207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:44.171454323 +0000 UTC m=+1609.203353262" watchObservedRunningTime="2026-10-30 14:15:44.175926207 +0000 UTC m=+1609.207825149"
Oct 30 14:15:44.487935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:44.487904    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:45.013833 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:45.013797    4530 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8530935_762e_43b0_bf08_3dd3149686be.slice/crio-3d50cdd2e8f9a757410d488a614f6a7dab1fc2f432b7ec2b8e824d3109e15cb0.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 14:15:45.032409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.032363    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" event={"ID":"daef6f1c-c355-4357-aa4c-f94ba2619dd9","Type":"ContainerStarted","Data":"e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035"}
Oct 30 14:15:45.034327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.034298    4530 generic.go:356] "Generic (PLEG): container finished" podID="e8530935-762e-43b0-bf08-3dd3149686be" containerID="3d50cdd2e8f9a757410d488a614f6a7dab1fc2f432b7ec2b8e824d3109e15cb0" exitCode=137
Oct 30 14:15:45.050357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.050326    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"]
Oct 30 14:15:45.064505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.064475    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f933cc-hck22"]
Oct 30 14:15:45.140799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.137588    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b7328ad-201c-41b8-b9c8-a32423392f40" path="/var/lib/kubelet/pods/7b7328ad-201c-41b8-b9c8-a32423392f40/volumes"
Oct 30 14:15:45.228356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.228327    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-243730"
Oct 30 14:15:45.244095 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:45.244060    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-243730\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e8530935-762e-43b0-bf08-3dd3149686be" pod="opendatahub/test-curl-9603-243730"
Oct 30 14:15:45.357764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.357730    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8530935-762e-43b0-bf08-3dd3149686be-kube-api-access-b8jph\" (UniqueName: \"kubernetes.io/projected/e8530935-762e-43b0-bf08-3dd3149686be-kube-api-access-b8jph\") pod \"e8530935-762e-43b0-bf08-3dd3149686be\" (UID: \"e8530935-762e-43b0-bf08-3dd3149686be\") "
Oct 30 14:15:45.360006 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.359979    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8530935-762e-43b0-bf08-3dd3149686be-kube-api-access-b8jph" pod "e8530935-762e-43b0-bf08-3dd3149686be" (UID: "e8530935-762e-43b0-bf08-3dd3149686be"). InnerVolumeSpecName "kube-api-access-b8jph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:15:45.497630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.471065    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8jph\" (UniqueName: \"kubernetes.io/projected/e8530935-762e-43b0-bf08-3dd3149686be-kube-api-access-b8jph\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:15:45.844567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.841961    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:15:45.844567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.844452    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:15:45.868571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.860492    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 14:15:45.868571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:45.860570    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 14:15:46.090482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:46.090447    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-243730"
Oct 30 14:15:46.098168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:46.098091    4530 scope.go:122] "RemoveContainer" containerID="3d50cdd2e8f9a757410d488a614f6a7dab1fc2f432b7ec2b8e824d3109e15cb0"
Oct 30 14:15:46.119277 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:46.119227    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-243730\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e8530935-762e-43b0-bf08-3dd3149686be" pod="opendatahub/test-curl-9603-243730"
Oct 30 14:15:46.137143 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:46.137099    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-243730\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e8530935-762e-43b0-bf08-3dd3149686be" pod="opendatahub/test-curl-9603-243730"
Oct 30 14:15:46.745289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:46.745248    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 14:15:46.745463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:46.745305    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 14:15:47.245565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:47.243205    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8530935-762e-43b0-bf08-3dd3149686be" path="/var/lib/kubelet/pods/e8530935-762e-43b0-bf08-3dd3149686be/volumes"
Oct 30 14:15:47.745856 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:47.745820    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 14:15:47.746007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:47.745887    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 14:15:48.746288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:48.746255    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 14:15:48.746702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:48.746324    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 14:15:49.745886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:49.745851    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 14:15:49.746059 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:49.745908    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 14:15:50.745391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:50.745361    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 14:15:50.745837 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:50.745413    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 14:15:51.802564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:51.799891    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:15:51.803069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:51.802770    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:15:52.488190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:52.488153    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:52.500928 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:52.500902    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:53.041745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:53.041715    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"]
Oct 30 14:15:53.055731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:53.055705    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6824b7ba-ngckn"]
Oct 30 14:15:53.118212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:53.118182    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a63741b-c431-4056-94d2-a996861a0388" path="/var/lib/kubelet/pods/4a63741b-c431-4056-94d2-a996861a0388/volumes"
Oct 30 14:15:53.356343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:53.356316    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:15:55.057341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:55.057307    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9603-568a02"]
Oct 30 14:15:55.057844 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:55.057487    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9603-568a02" podUID="e51dd13e-af5e-49d9-a333-414e5f493923" containerName="test-curl-9603-568a02" containerID="cri-o://85574c8bef999895e7935a21b2b6d9cbf6264ce53672a0a3fac185c28ef4ab1b" gracePeriod=2
Oct 30 14:15:55.065772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:55.065738    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9603-568a02"]
Oct 30 14:15:55.073015 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:55.072979    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-568a02\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e51dd13e-af5e-49d9-a333-414e5f493923" pod="opendatahub/test-curl-9603-568a02"
Oct 30 14:15:55.132377 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:55.132334    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-568a02\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e51dd13e-af5e-49d9-a333-414e5f493923" pod="opendatahub/test-curl-9603-568a02"
Oct 30 14:15:56.042968 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.042936    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"]
Oct 30 14:15:56.048613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.048585    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a976ca-z99px"]
Oct 30 14:15:56.078440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.078409    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"]
Oct 30 14:15:56.079667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.079639    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e51dd13e-af5e-49d9-a333-414e5f493923" containerName="test-curl-9603-568a02"
Oct 30 14:15:56.079667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.079665    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51dd13e-af5e-49d9-a333-414e5f493923" containerName="test-curl-9603-568a02"
Oct 30 14:15:56.079891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.079875    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51dd13e-af5e-49d9-a333-414e5f493923" containerName="test-curl-9603-568a02"
Oct 30 14:15:56.085730 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.085400    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.107520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.107491    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-01955278-openshift-default-dockercfg-fg28k\""
Oct 30 14:15:56.109193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.109168    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"]
Oct 30 14:15:56.148351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.148302    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-credential-socket\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.148534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.148430    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-envoy\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.148534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.148485    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-socket\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.148731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.148701    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-data\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.148816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.148744    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-certs\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.148892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.148830    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-podinfo\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.148892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.148874    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-token\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.149016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.148924    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42dedd41-345f-4dd6-8519-fa7587c4a383-istiod-ca-cert\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.149070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.149028    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz2l5\" (UniqueName: \"kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-kube-api-access-tz2l5\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.289567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.280323    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-socket\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.289567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.280536    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-data\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.289567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.280702    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-certs\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.289567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.280803    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-podinfo\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.289567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.280852    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-token\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.289567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.280916    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42dedd41-345f-4dd6-8519-fa7587c4a383-istiod-ca-cert\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.289567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.281037    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz2l5\" (UniqueName: \"kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-kube-api-access-tz2l5\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.289567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.281168    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-credential-socket\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.289567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.281221    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-envoy\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.291385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.290449    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-certs\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.291385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.290953    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42dedd41-345f-4dd6-8519-fa7587c4a383-istiod-ca-cert\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.291877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.291851    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-socket\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.292145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.292123    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-data\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.331615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.296292    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-credential-socket\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.422640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.372122    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-podinfo\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.422920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.392822    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-envoy\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.509568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.505384    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz2l5\" (UniqueName: \"kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-kube-api-access-tz2l5\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.588269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.588152    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-token\") pod \"e2e-mt-01955278-openshift-default-56c64d6797-9hfnc\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") " pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:56.708227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:56.707923    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:57.330661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.330615    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:15:57.331380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.331350    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:15:57.331570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.331526    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:15:57.350209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.350129    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73d6c78-38f5-4172-a216-f511ac326c95" path="/var/lib/kubelet/pods/c73d6c78-38f5-4172-a216-f511ac326c95/volumes"
Oct 30 14:15:57.403623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.403535    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"]
Oct 30 14:15:57.597068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.596720    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" event={"ID":"42dedd41-345f-4dd6-8519-fa7587c4a383","Type":"ContainerStarted","Data":"e475cb9e369d7116ad6ddac404d7a1a5ca626df82a91b67afe04079a0d2fd7ae"}
Oct 30 14:15:57.597068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.596786    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" event={"ID":"42dedd41-345f-4dd6-8519-fa7587c4a383","Type":"ContainerStarted","Data":"af6d291b18c805407214d77957e02abab0b440e8d09cd261e886c2fa9bb79555"}
Oct 30 14:15:57.599045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.599020    4530 generic.go:356] "Generic (PLEG): container finished" podID="e51dd13e-af5e-49d9-a333-414e5f493923" containerID="85574c8bef999895e7935a21b2b6d9cbf6264ce53672a0a3fac185c28ef4ab1b" exitCode=137
Oct 30 14:15:57.599250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.599237    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b382a03735dc932163c857052273820e4600fa22e0ad3d941ebaf47f219fbfa"
Oct 30 14:15:57.628307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.628278    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-568a02"
Oct 30 14:15:57.669565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.659973    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" podStartSLOduration=1.659951769 podStartE2EDuration="1.659951769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:57.658355819 +0000 UTC m=+1622.690254784" watchObservedRunningTime="2026-10-30 14:15:57.659951769 +0000 UTC m=+1622.691850734"
Oct 30 14:15:57.717647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.717605    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:57.780722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.780619    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51dd13e-af5e-49d9-a333-414e5f493923-kube-api-access-7cpt2\" (UniqueName: \"kubernetes.io/projected/e51dd13e-af5e-49d9-a333-414e5f493923-kube-api-access-7cpt2\") pod \"e51dd13e-af5e-49d9-a333-414e5f493923\" (UID: \"e51dd13e-af5e-49d9-a333-414e5f493923\") "
Oct 30 14:15:57.821492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.821452    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51dd13e-af5e-49d9-a333-414e5f493923-kube-api-access-7cpt2" pod "e51dd13e-af5e-49d9-a333-414e5f493923" (UID: "e51dd13e-af5e-49d9-a333-414e5f493923"). InnerVolumeSpecName "kube-api-access-7cpt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:15:57.911193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:57.911148    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cpt2\" (UniqueName: \"kubernetes.io/projected/e51dd13e-af5e-49d9-a333-414e5f493923-kube-api-access-7cpt2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:15:58.608025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:58.607994    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-568a02"
Oct 30 14:15:58.632020 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:58.631984    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-568a02\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e51dd13e-af5e-49d9-a333-414e5f493923" pod="opendatahub/test-curl-9603-568a02"
Oct 30 14:15:58.718633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:58.718603    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:58.734063 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:15:58.734030    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-568a02\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e51dd13e-af5e-49d9-a333-414e5f493923" pod="opendatahub/test-curl-9603-568a02"
Oct 30 14:15:59.116805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:59.116773    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51dd13e-af5e-49d9-a333-414e5f493923" path="/var/lib/kubelet/pods/e51dd13e-af5e-49d9-a333-414e5f493923/volumes"
Oct 30 14:15:59.617726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:59.617694    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:59.618914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:59.618888    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:15:59.924201 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:59.924127    4530 scope.go:122] "RemoveContainer" containerID="98cbb65a76be90d7cb7ed3010b82d109924e34f55fce20d34300dfd0f6f1f7e6"
Oct 30 14:15:59.937201 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:59.937176    4530 scope.go:122] "RemoveContainer" containerID="03d41d11e91f1097a8143120eb5b91e28323511ae8d66d445f12ef24678f43bb"
Oct 30 14:15:59.948466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:59.948443    4530 scope.go:122] "RemoveContainer" containerID="6a95e04df92e872810215699332ad2c63890104525e62c84ffd484dd6a10a1d3"
Oct 30 14:15:59.959338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:15:59.959312    4530 scope.go:122] "RemoveContainer" containerID="9015b8779589504dff52b7d6dcbaee0418b98e1f9e859adc69ba3adaf93fb11e"
Oct 30 14:16:03.041782 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:03.041742    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:16:05.255971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:05.255935    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9603-e34615"]
Oct 30 14:16:05.262847 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:05.262822    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-e34615"
Oct 30 14:16:05.278915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:05.278888    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9603-e34615"]
Oct 30 14:16:05.336452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:05.336423    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q95l7\""
Oct 30 14:16:05.454382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:05.454345    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9blp\" (UniqueName: \"kubernetes.io/projected/007a56ed-0956-43a8-935a-65b072a89e0a-kube-api-access-s9blp\") pod \"test-curl-9603-e34615\" (UID: \"007a56ed-0956-43a8-935a-65b072a89e0a\") " pod="opendatahub/test-curl-9603-e34615"
Oct 30 14:16:05.555603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:05.555567    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9blp\" (UniqueName: \"kubernetes.io/projected/007a56ed-0956-43a8-935a-65b072a89e0a-kube-api-access-s9blp\") pod \"test-curl-9603-e34615\" (UID: \"007a56ed-0956-43a8-935a-65b072a89e0a\") " pod="opendatahub/test-curl-9603-e34615"
Oct 30 14:16:05.582262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:05.582234    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9blp\" (UniqueName: \"kubernetes.io/projected/007a56ed-0956-43a8-935a-65b072a89e0a-kube-api-access-s9blp\") pod \"test-curl-9603-e34615\" (UID: \"007a56ed-0956-43a8-935a-65b072a89e0a\") " pod="opendatahub/test-curl-9603-e34615"
Oct 30 14:16:05.880371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:05.880276    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-e34615"
Oct 30 14:16:06.213349 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:16:06.213315    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod007a56ed_0956_43a8_935a_65b072a89e0a.slice/crio-6cc6af30e067b42de099411321d9abb58950382ffae3e97dc39c37c7de208982 WatchSource:0}: Error finding container 6cc6af30e067b42de099411321d9abb58950382ffae3e97dc39c37c7de208982: Status 404 returned error can't find the container with id 6cc6af30e067b42de099411321d9abb58950382ffae3e97dc39c37c7de208982
Oct 30 14:16:06.216781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:06.216759    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9603-e34615"]
Oct 30 14:16:06.841207 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:06.841166    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9603-e34615" event={"ID":"007a56ed-0956-43a8-935a-65b072a89e0a","Type":"ContainerStarted","Data":"cbf98281f8938ece15e28d6357fbcf4ba06b4029aa2d7805ff97f7084f35a639"}
Oct 30 14:16:06.841207 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:06.841210    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9603-e34615" event={"ID":"007a56ed-0956-43a8-935a-65b072a89e0a","Type":"ContainerStarted","Data":"6cc6af30e067b42de099411321d9abb58950382ffae3e97dc39c37c7de208982"}
Oct 30 14:16:06.884345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:06.884295    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9603-e34615" podStartSLOduration=1.884283261 podStartE2EDuration="1.884283261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:16:06.884073497 +0000 UTC m=+1631.915972437" watchObservedRunningTime="2026-10-30 14:16:06.884283261 +0000 UTC m=+1631.916182200"
Oct 30 14:16:07.725201 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:07.725167    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9603-e34615"]
Oct 30 14:16:07.734326 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:07.734292    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9603-e34615"]
Oct 30 14:16:07.889669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:07.888145    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9603-4db048"]
Oct 30 14:16:07.890182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:07.889936    4530 cpu_manager.go:414] "containerMap: removing container" podUID="007a56ed-0956-43a8-935a-65b072a89e0a" containerName="test-curl-9603-e34615"
Oct 30 14:16:07.890182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:07.889962    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007a56ed-0956-43a8-935a-65b072a89e0a" containerName="test-curl-9603-e34615"
Oct 30 14:16:07.890292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:07.890255    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="007a56ed-0956-43a8-935a-65b072a89e0a" containerName="test-curl-9603-e34615"
Oct 30 14:16:07.933614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:07.931324    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-4db048"
Oct 30 14:16:07.972280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:07.972244    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9603-4db048"]
Oct 30 14:16:08.040744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:08.040654    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx666\" (UniqueName: \"kubernetes.io/projected/e56997e4-1701-4fd2-8ddb-b12da63a113e-kube-api-access-hx666\") pod \"test-curl-9603-4db048\" (UID: \"e56997e4-1701-4fd2-8ddb-b12da63a113e\") " pod="opendatahub/test-curl-9603-4db048"
Oct 30 14:16:08.142512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:08.142474    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx666\" (UniqueName: \"kubernetes.io/projected/e56997e4-1701-4fd2-8ddb-b12da63a113e-kube-api-access-hx666\") pod \"test-curl-9603-4db048\" (UID: \"e56997e4-1701-4fd2-8ddb-b12da63a113e\") " pod="opendatahub/test-curl-9603-4db048"
Oct 30 14:16:08.169218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:08.169194    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx666\" (UniqueName: \"kubernetes.io/projected/e56997e4-1701-4fd2-8ddb-b12da63a113e-kube-api-access-hx666\") pod \"test-curl-9603-4db048\" (UID: \"e56997e4-1701-4fd2-8ddb-b12da63a113e\") " pod="opendatahub/test-curl-9603-4db048"
Oct 30 14:16:08.287811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:08.287780    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-4db048"
Oct 30 14:16:08.593266 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:16:08.593236    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode56997e4_1701_4fd2_8ddb_b12da63a113e.slice/crio-84b028f5ef5fccc4e6ca4836358c0532e250b57c6327e5cdf5bd81f7631cf78e WatchSource:0}: Error finding container 84b028f5ef5fccc4e6ca4836358c0532e250b57c6327e5cdf5bd81f7631cf78e: Status 404 returned error can't find the container with id 84b028f5ef5fccc4e6ca4836358c0532e250b57c6327e5cdf5bd81f7631cf78e
Oct 30 14:16:08.597330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:08.597301    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9603-4db048"]
Oct 30 14:16:08.980529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:08.980437    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9603-4db048" event={"ID":"e56997e4-1701-4fd2-8ddb-b12da63a113e","Type":"ContainerStarted","Data":"60cf8cb5c00ddf9ca17fdac8b288e67fc04652b7d254b32decb3a8dff63c7100"}
Oct 30 14:16:08.980529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:08.980490    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9603-4db048" event={"ID":"e56997e4-1701-4fd2-8ddb-b12da63a113e","Type":"ContainerStarted","Data":"84b028f5ef5fccc4e6ca4836358c0532e250b57c6327e5cdf5bd81f7631cf78e"}
Oct 30 14:16:08.981191 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:08.980522    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9603-e34615" podUID="007a56ed-0956-43a8-935a-65b072a89e0a" containerName="test-curl-9603-e34615" containerID="cri-o://cbf98281f8938ece15e28d6357fbcf4ba06b4029aa2d7805ff97f7084f35a639" gracePeriod=2
Oct 30 14:16:08.996015 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:16:08.995979    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-e34615\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="007a56ed-0956-43a8-935a-65b072a89e0a" pod="opendatahub/test-curl-9603-e34615"
Oct 30 14:16:09.045351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:09.045306    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9603-4db048" podStartSLOduration=2.045292801 podStartE2EDuration="2.045292801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:16:09.03903891 +0000 UTC m=+1634.070937853" watchObservedRunningTime="2026-10-30 14:16:09.045292801 +0000 UTC m=+1634.077191737"
Oct 30 14:16:09.054694 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:16:09.054671    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-e34615\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="007a56ed-0956-43a8-935a-65b072a89e0a" pod="opendatahub/test-curl-9603-e34615"
Oct 30 14:16:11.644914 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:16:11.644591    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:16:11.673192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:11.673163    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-e34615"
Oct 30 14:16:11.762961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:11.762880    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007a56ed-0956-43a8-935a-65b072a89e0a-kube-api-access-s9blp\" (UniqueName: \"kubernetes.io/projected/007a56ed-0956-43a8-935a-65b072a89e0a-kube-api-access-s9blp\") pod \"007a56ed-0956-43a8-935a-65b072a89e0a\" (UID: \"007a56ed-0956-43a8-935a-65b072a89e0a\") "
Oct 30 14:16:11.765145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:11.765111    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007a56ed-0956-43a8-935a-65b072a89e0a-kube-api-access-s9blp" pod "007a56ed-0956-43a8-935a-65b072a89e0a" (UID: "007a56ed-0956-43a8-935a-65b072a89e0a"). InnerVolumeSpecName "kube-api-access-s9blp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:16:11.887274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:11.886168    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9blp\" (UniqueName: \"kubernetes.io/projected/007a56ed-0956-43a8-935a-65b072a89e0a-kube-api-access-s9blp\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:12.016802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:12.016719    4530 generic.go:356] "Generic (PLEG): container finished" podID="007a56ed-0956-43a8-935a-65b072a89e0a" containerID="cbf98281f8938ece15e28d6357fbcf4ba06b4029aa2d7805ff97f7084f35a639" exitCode=137
Oct 30 14:16:12.016802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:12.016766    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-e34615"
Oct 30 14:16:12.016985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:12.016824    4530 scope.go:122] "RemoveContainer" containerID="cbf98281f8938ece15e28d6357fbcf4ba06b4029aa2d7805ff97f7084f35a639"
Oct 30 14:16:12.028704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:12.028687    4530 scope.go:122] "RemoveContainer" containerID="cbf98281f8938ece15e28d6357fbcf4ba06b4029aa2d7805ff97f7084f35a639"
Oct 30 14:16:12.029013 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:16:12.028993    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf98281f8938ece15e28d6357fbcf4ba06b4029aa2d7805ff97f7084f35a639\": container with ID starting with cbf98281f8938ece15e28d6357fbcf4ba06b4029aa2d7805ff97f7084f35a639 not found: ID does not exist" containerID="cbf98281f8938ece15e28d6357fbcf4ba06b4029aa2d7805ff97f7084f35a639"
Oct 30 14:16:12.029074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:12.029023    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf98281f8938ece15e28d6357fbcf4ba06b4029aa2d7805ff97f7084f35a639"} err="failed to get container status \"cbf98281f8938ece15e28d6357fbcf4ba06b4029aa2d7805ff97f7084f35a639\": rpc error: code = NotFound desc = could not find container \"cbf98281f8938ece15e28d6357fbcf4ba06b4029aa2d7805ff97f7084f35a639\": container with ID starting with cbf98281f8938ece15e28d6357fbcf4ba06b4029aa2d7805ff97f7084f35a639 not found: ID does not exist"
Oct 30 14:16:12.043369 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:16:12.043342    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-e34615\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="007a56ed-0956-43a8-935a-65b072a89e0a" pod="opendatahub/test-curl-9603-e34615"
Oct 30 14:16:13.116902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:13.116864    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007a56ed-0956-43a8-935a-65b072a89e0a" path="/var/lib/kubelet/pods/007a56ed-0956-43a8-935a-65b072a89e0a/volumes"
Oct 30 14:16:15.038206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:15.038169    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"]
Oct 30 14:16:15.055320 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:15.055292    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56011259-8cjsm"]
Oct 30 14:16:15.117227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:15.117198    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7" path="/var/lib/kubelet/pods/cc1fbf64-5e02-43ef-a805-7b7d0aa19ce7/volumes"
Oct 30 14:16:19.037504 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:19.037470    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"]
Oct 30 14:16:19.051286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:19.051255    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8dcb50-58x5w"]
Oct 30 14:16:19.116588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:19.116557    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ca9125-32d2-4407-b359-7f3d56602f59" path="/var/lib/kubelet/pods/b9ca9125-32d2-4407-b359-7f3d56602f59/volumes"
Oct 30 14:16:22.038366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:22.038333    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"]
Oct 30 14:16:22.052187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:22.052162    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f9ac612-m2mzc"]
Oct 30 14:16:23.117887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:23.117851    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8422e767-014d-4ace-8622-17530ce03eee" path="/var/lib/kubelet/pods/8422e767-014d-4ace-8622-17530ce03eee/volumes"
Oct 30 14:16:26.087072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:26.086997    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"]
Oct 30 14:16:26.115525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:26.115492    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-476cdbdd-6hhhw"]
Oct 30 14:16:26.139713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:26.137929    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"]
Oct 30 14:16:26.139713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:26.138169    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerName="istio-proxy" containerID="cri-o://9e1028d0d75b9cdb5ed3208f264c354558f5ca8ac8b8192e29e8f6f126a37f9f" gracePeriod=30
Oct 30 14:16:26.445633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:26.440651    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"]
Oct 30 14:16:26.445633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:26.440933    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy" containerID="cri-o://b8a69ae8796b70107b66408024a1ce4e2099ac2d2b6b14ff987fd763214f1a34" gracePeriod=30
Oct 30 14:16:26.558565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:26.558225    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"]
Oct 30 14:16:26.558565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:26.558467    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" podUID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" containerName="istio-proxy" containerID="cri-o://4407ac55a8048fb70d2ac3e32039152b6b76d39aa8edb1f00deede39942054a9" gracePeriod=30
Oct 30 14:16:26.726640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:26.723586    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"]
Oct 30 14:16:26.726640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:26.723850    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy" containerID="cri-o://69262667ecdf8d545bdb6167e6cb61256031486673cff2095bc82dedf9b9b17a" gracePeriod=30
Oct 30 14:16:27.167483 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:27.167444    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2bef961-40fb-4eb6-9d68-faf4caa9faee" path="/var/lib/kubelet/pods/b2bef961-40fb-4eb6-9d68-faf4caa9faee/volumes"
Oct 30 14:16:27.423700 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:27.422614    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr" podUID="91d82203-dd8e-4e1f-ad76-0403aea6eaca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": context deadline exceeded"
Oct 30 14:16:27.444883 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:27.443622    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:16:27.454569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:27.452743    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded"
Oct 30 14:16:27.458563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:27.457614    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:16:27.497565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:27.494617    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:16:28.149206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.148953    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/health\": context deadline exceeded"
Oct 30 14:16:28.441632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.434091    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr" podUID="91d82203-dd8e-4e1f-ad76-0403aea6eaca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:16:28.441632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.435687    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:16:28.465242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.465195    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded"
Oct 30 14:16:28.702067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.466283    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 14:16:28.702067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.695018    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:16:28.702067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.695079    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:16:28.702067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.491961    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:16:28.702067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.666986    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Oct 30 14:16:28.702067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.697930    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:16:28.702067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.698036    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:16:28.851149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.851115    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:16:28.948740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.948707    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:16:28.949126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.949105    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:16:28.949363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:28.949347    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:16:29.132981 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:29.101102    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:16:31.919388 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:31.919322    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"]
Oct 30 14:16:31.933230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:31.932497    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:31.933392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:31.933344    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"]
Oct 30 14:16:31.956564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:31.951187    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9dba4af9-openshift-default-dockercfg-qf6sh\""
Oct 30 14:16:32.063511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.063474    4530 generic.go:356] "Generic (PLEG): container finished" podID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerID="69262667ecdf8d545bdb6167e6cb61256031486673cff2095bc82dedf9b9b17a" exitCode=0
Oct 30 14:16:32.063899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.063703    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" event={"ID":"e58017dd-1638-4df7-a7f9-dffd6db076f6","Type":"ContainerDied","Data":"69262667ecdf8d545bdb6167e6cb61256031486673cff2095bc82dedf9b9b17a"}
Oct 30 14:16:32.067420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.067390    4530 generic.go:356] "Generic (PLEG): container finished" podID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerID="9e1028d0d75b9cdb5ed3208f264c354558f5ca8ac8b8192e29e8f6f126a37f9f" exitCode=0
Oct 30 14:16:32.067564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.067519    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" event={"ID":"97052caf-e138-4d01-8b20-a8cade8b7a3d","Type":"ContainerDied","Data":"9e1028d0d75b9cdb5ed3208f264c354558f5ca8ac8b8192e29e8f6f126a37f9f"}
Oct 30 14:16:32.067616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.067593    4530 scope.go:122] "RemoveContainer" containerID="cedcf25d9c3d52ec664e14f379225288a89a2da59acd8bfe1c36c5b4c6a2f3b2"
Oct 30 14:16:32.069971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.069931    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-credential-socket\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.070900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.070880    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-podinfo\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.071107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.071091    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-socket\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.071247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.071233    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istiod-ca-cert\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.071352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.071339    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-envoy\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.071498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.071482    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-data\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.071690 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.071672    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-token\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.071850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.071834    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-certs\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.072099 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.072080    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlj2p\" (UniqueName: \"kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-kube-api-access-jlj2p\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.073885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.073849    4530 generic.go:356] "Generic (PLEG): container finished" podID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerID="b8a69ae8796b70107b66408024a1ce4e2099ac2d2b6b14ff987fd763214f1a34" exitCode=0
Oct 30 14:16:32.076911 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.076887    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" event={"ID":"54eb3905-9d69-4cf9-8cc8-a279635e8183","Type":"ContainerDied","Data":"b8a69ae8796b70107b66408024a1ce4e2099ac2d2b6b14ff987fd763214f1a34"}
Oct 30 14:16:32.115805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.115729    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" event={"ID":"48f881b2-0e66-4706-b9d1-4b1842b5d33c","Type":"ContainerDied","Data":"4407ac55a8048fb70d2ac3e32039152b6b76d39aa8edb1f00deede39942054a9"}
Oct 30 14:16:32.115989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.115829    4530 generic.go:356] "Generic (PLEG): container finished" podID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" containerID="4407ac55a8048fb70d2ac3e32039152b6b76d39aa8edb1f00deede39942054a9" exitCode=0
Oct 30 14:16:32.173435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.173399    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-podinfo\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.173644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.173443    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-socket\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.173644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.173502    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istiod-ca-cert\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.173644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.173526    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-envoy\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.173811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.173663    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-data\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.173811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.173743    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-token\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.173811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.173788    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-certs\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.174477 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.174457    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istiod-ca-cert\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.175730 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.175381    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-socket\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.175829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.175810    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-data\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.175961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.175941    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlj2p\" (UniqueName: \"kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-kube-api-access-jlj2p\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.176019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.176001    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-certs\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.176171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.176152    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-credential-socket\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.176466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.176448    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-credential-socket\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.179326 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.179299    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-podinfo\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.234642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.230032    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-envoy\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.241175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.237641    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-token\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.242641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.242505    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlj2p\" (UniqueName: \"kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-kube-api-access-jlj2p\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.334708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.334485    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:32.339634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.339606    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:16:32.362482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.362446    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:16:32.366113 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.366084    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:16:32.370598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.370577    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:16:32.387564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.380011    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-credential-socket\") pod \"97052caf-e138-4d01-8b20-a8cade8b7a3d\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") "
Oct 30 14:16:32.387564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.380071    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-envoy\") pod \"97052caf-e138-4d01-8b20-a8cade8b7a3d\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") "
Oct 30 14:16:32.387564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.380171    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-data\") pod \"97052caf-e138-4d01-8b20-a8cade8b7a3d\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") "
Oct 30 14:16:32.387564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.380209    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-kube-api-access-6sp28\" (UniqueName: \"kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-kube-api-access-6sp28\") pod \"97052caf-e138-4d01-8b20-a8cade8b7a3d\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") "
Oct 30 14:16:32.387564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.380277    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-certs\") pod \"97052caf-e138-4d01-8b20-a8cade8b7a3d\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") "
Oct 30 14:16:32.387564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.380410    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97052caf-e138-4d01-8b20-a8cade8b7a3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97052caf-e138-4d01-8b20-a8cade8b7a3d-istiod-ca-cert\") pod \"97052caf-e138-4d01-8b20-a8cade8b7a3d\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") "
Oct 30 14:16:32.387564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.380579    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-token\" (UniqueName: \"kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-token\") pod \"97052caf-e138-4d01-8b20-a8cade8b7a3d\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") "
Oct 30 14:16:32.387564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.380677    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-socket\") pod \"97052caf-e138-4d01-8b20-a8cade8b7a3d\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") "
Oct 30 14:16:32.387564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.380742    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-podinfo\") pod \"97052caf-e138-4d01-8b20-a8cade8b7a3d\" (UID: \"97052caf-e138-4d01-8b20-a8cade8b7a3d\") "
Oct 30 14:16:32.416001 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.412459    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97052caf-e138-4d01-8b20-a8cade8b7a3d-istiod-ca-cert" pod "97052caf-e138-4d01-8b20-a8cade8b7a3d" (UID: "97052caf-e138-4d01-8b20-a8cade8b7a3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:16:32.416001 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.412844    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-certs" pod "97052caf-e138-4d01-8b20-a8cade8b7a3d" (UID: "97052caf-e138-4d01-8b20-a8cade8b7a3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.416001 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.413851    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-credential-socket" pod "97052caf-e138-4d01-8b20-a8cade8b7a3d" (UID: "97052caf-e138-4d01-8b20-a8cade8b7a3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.416001 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.414068    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-data" pod "97052caf-e138-4d01-8b20-a8cade8b7a3d" (UID: "97052caf-e138-4d01-8b20-a8cade8b7a3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.416001 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.414666    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-socket" pod "97052caf-e138-4d01-8b20-a8cade8b7a3d" (UID: "97052caf-e138-4d01-8b20-a8cade8b7a3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.419606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.417057    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.419606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.417075    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.419606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.417085    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.419606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.417094    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.419606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.417102    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97052caf-e138-4d01-8b20-a8cade8b7a3d-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.425015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.424974    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-podinfo" pod "97052caf-e138-4d01-8b20-a8cade8b7a3d" (UID: "97052caf-e138-4d01-8b20-a8cade8b7a3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:16:32.426279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.425134    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-envoy" pod "97052caf-e138-4d01-8b20-a8cade8b7a3d" (UID: "97052caf-e138-4d01-8b20-a8cade8b7a3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.426279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.425313    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-kube-api-access-6sp28" pod "97052caf-e138-4d01-8b20-a8cade8b7a3d" (UID: "97052caf-e138-4d01-8b20-a8cade8b7a3d"). InnerVolumeSpecName "kube-api-access-6sp28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:16:32.427473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.427447    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-token" pod "97052caf-e138-4d01-8b20-a8cade8b7a3d" (UID: "97052caf-e138-4d01-8b20-a8cade8b7a3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:16:32.518505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.518421    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-podinfo\") pod \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") "
Oct 30 14:16:32.519054 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.518803    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-data\") pod \"e58017dd-1638-4df7-a7f9-dffd6db076f6\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") "
Oct 30 14:16:32.519242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.519022    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-data" pod "e58017dd-1638-4df7-a7f9-dffd6db076f6" (UID: "e58017dd-1638-4df7-a7f9-dffd6db076f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.519367 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.519355    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-kube-api-access-9vt7p\" (UniqueName: \"kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-kube-api-access-9vt7p\") pod \"e58017dd-1638-4df7-a7f9-dffd6db076f6\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") "
Oct 30 14:16:32.520314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.519858    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-token\" (UniqueName: \"kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-token\") pod \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") "
Oct 30 14:16:32.522222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.522199    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-podinfo\") pod \"54eb3905-9d69-4cf9-8cc8-a279635e8183\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") "
Oct 30 14:16:32.523567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.523452    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-token" pod "48f881b2-0e66-4706-b9d1-4b1842b5d33c" (UID: "48f881b2-0e66-4706-b9d1-4b1842b5d33c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:16:32.524000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.523977    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-envoy\") pod \"54eb3905-9d69-4cf9-8cc8-a279635e8183\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526008    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istiod-ca-cert\") pod \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526049    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-kube-api-access-ssmdc\" (UniqueName: \"kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-kube-api-access-ssmdc\") pod \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526101    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-podinfo\") pod \"e58017dd-1638-4df7-a7f9-dffd6db076f6\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526171    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-socket\") pod \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526229    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-socket\") pod \"54eb3905-9d69-4cf9-8cc8-a279635e8183\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526276    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-certs\") pod \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526326    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-kube-api-access-v6mwr\" (UniqueName: \"kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-kube-api-access-v6mwr\") pod \"54eb3905-9d69-4cf9-8cc8-a279635e8183\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526357    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-envoy\") pod \"e58017dd-1638-4df7-a7f9-dffd6db076f6\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526401    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-data\") pod \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526437    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-envoy\") pod \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526463    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-token\" (UniqueName: \"kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-token\") pod \"e58017dd-1638-4df7-a7f9-dffd6db076f6\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526492    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-socket\") pod \"e58017dd-1638-4df7-a7f9-dffd6db076f6\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526525    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-credential-socket\") pod \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\" (UID: \"48f881b2-0e66-4706-b9d1-4b1842b5d33c\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526920    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-certs\") pod \"e58017dd-1638-4df7-a7f9-dffd6db076f6\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.526980    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e58017dd-1638-4df7-a7f9-dffd6db076f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58017dd-1638-4df7-a7f9-dffd6db076f6-istiod-ca-cert\") pod \"e58017dd-1638-4df7-a7f9-dffd6db076f6\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.527032    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-token\" (UniqueName: \"kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-token\") pod \"54eb3905-9d69-4cf9-8cc8-a279635e8183\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.527107    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-socket" pod "48f881b2-0e66-4706-b9d1-4b1842b5d33c" (UID: "48f881b2-0e66-4706-b9d1-4b1842b5d33c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.527141    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-credential-socket\") pod \"e58017dd-1638-4df7-a7f9-dffd6db076f6\" (UID: \"e58017dd-1638-4df7-a7f9-dffd6db076f6\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.527233    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-data\") pod \"54eb3905-9d69-4cf9-8cc8-a279635e8183\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.527284    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-credential-socket\") pod \"54eb3905-9d69-4cf9-8cc8-a279635e8183\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.527383    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-certs\") pod \"54eb3905-9d69-4cf9-8cc8-a279635e8183\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.527448    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54eb3905-9d69-4cf9-8cc8-a279635e8183-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54eb3905-9d69-4cf9-8cc8-a279635e8183-istiod-ca-cert\") pod \"54eb3905-9d69-4cf9-8cc8-a279635e8183\" (UID: \"54eb3905-9d69-4cf9-8cc8-a279635e8183\") "
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.527607    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istiod-ca-cert" pod "48f881b2-0e66-4706-b9d1-4b1842b5d33c" (UID: "48f881b2-0e66-4706-b9d1-4b1842b5d33c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.527923    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-data" pod "48f881b2-0e66-4706-b9d1-4b1842b5d33c" (UID: "48f881b2-0e66-4706-b9d1-4b1842b5d33c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.527956    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-socket" pod "54eb3905-9d69-4cf9-8cc8-a279635e8183" (UID: "54eb3905-9d69-4cf9-8cc8-a279635e8183"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528309    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54eb3905-9d69-4cf9-8cc8-a279635e8183-istiod-ca-cert" pod "54eb3905-9d69-4cf9-8cc8-a279635e8183" (UID: "54eb3905-9d69-4cf9-8cc8-a279635e8183"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528526    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-certs" pod "48f881b2-0e66-4706-b9d1-4b1842b5d33c" (UID: "48f881b2-0e66-4706-b9d1-4b1842b5d33c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528809    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528834    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528849    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528865    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528888    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528903    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528917    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528930    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528944    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97052caf-e138-4d01-8b20-a8cade8b7a3d-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528958    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sp28\" (UniqueName: \"kubernetes.io/projected/97052caf-e138-4d01-8b20-a8cade8b7a3d-kube-api-access-6sp28\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528972    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54eb3905-9d69-4cf9-8cc8-a279635e8183-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.528985    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.529206    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-credential-socket" pod "e58017dd-1638-4df7-a7f9-dffd6db076f6" (UID: "e58017dd-1638-4df7-a7f9-dffd6db076f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.529384    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-data" pod "54eb3905-9d69-4cf9-8cc8-a279635e8183" (UID: "54eb3905-9d69-4cf9-8cc8-a279635e8183"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.529487    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-kube-api-access-v6mwr" pod "54eb3905-9d69-4cf9-8cc8-a279635e8183" (UID: "54eb3905-9d69-4cf9-8cc8-a279635e8183"). InnerVolumeSpecName "kube-api-access-v6mwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.529591    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-credential-socket" pod "54eb3905-9d69-4cf9-8cc8-a279635e8183" (UID: "54eb3905-9d69-4cf9-8cc8-a279635e8183"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.529733    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-certs" pod "54eb3905-9d69-4cf9-8cc8-a279635e8183" (UID: "54eb3905-9d69-4cf9-8cc8-a279635e8183"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.529798    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-credential-socket" pod "48f881b2-0e66-4706-b9d1-4b1842b5d33c" (UID: "48f881b2-0e66-4706-b9d1-4b1842b5d33c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.529975    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-socket" pod "e58017dd-1638-4df7-a7f9-dffd6db076f6" (UID: "e58017dd-1638-4df7-a7f9-dffd6db076f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.530035    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-certs" pod "e58017dd-1638-4df7-a7f9-dffd6db076f6" (UID: "e58017dd-1638-4df7-a7f9-dffd6db076f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.530286    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58017dd-1638-4df7-a7f9-dffd6db076f6-istiod-ca-cert" pod "e58017dd-1638-4df7-a7f9-dffd6db076f6" (UID: "e58017dd-1638-4df7-a7f9-dffd6db076f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.530349    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-envoy" pod "54eb3905-9d69-4cf9-8cc8-a279635e8183" (UID: "54eb3905-9d69-4cf9-8cc8-a279635e8183"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.530356    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-kube-api-access-9vt7p" pod "e58017dd-1638-4df7-a7f9-dffd6db076f6" (UID: "e58017dd-1638-4df7-a7f9-dffd6db076f6"). InnerVolumeSpecName "kube-api-access-9vt7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.530398    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-podinfo" pod "48f881b2-0e66-4706-b9d1-4b1842b5d33c" (UID: "48f881b2-0e66-4706-b9d1-4b1842b5d33c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:16:32.530463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.530421    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-podinfo" pod "54eb3905-9d69-4cf9-8cc8-a279635e8183" (UID: "54eb3905-9d69-4cf9-8cc8-a279635e8183"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:16:32.533381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.530791    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-podinfo" pod "e58017dd-1638-4df7-a7f9-dffd6db076f6" (UID: "e58017dd-1638-4df7-a7f9-dffd6db076f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:16:32.533381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.532592    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-kube-api-access-ssmdc" pod "48f881b2-0e66-4706-b9d1-4b1842b5d33c" (UID: "48f881b2-0e66-4706-b9d1-4b1842b5d33c"). InnerVolumeSpecName "kube-api-access-ssmdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:16:32.533381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.532791    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-token" pod "e58017dd-1638-4df7-a7f9-dffd6db076f6" (UID: "e58017dd-1638-4df7-a7f9-dffd6db076f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:16:32.533381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.532902    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-token" pod "54eb3905-9d69-4cf9-8cc8-a279635e8183" (UID: "54eb3905-9d69-4cf9-8cc8-a279635e8183"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:16:32.533381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.533374    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-envoy" pod "e58017dd-1638-4df7-a7f9-dffd6db076f6" (UID: "e58017dd-1638-4df7-a7f9-dffd6db076f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.534338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.534310    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-envoy" pod "48f881b2-0e66-4706-b9d1-4b1842b5d33c" (UID: "48f881b2-0e66-4706-b9d1-4b1842b5d33c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:16:32.630419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630383    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630414    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630425    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssmdc\" (UniqueName: \"kubernetes.io/projected/48f881b2-0e66-4706-b9d1-4b1842b5d33c-kube-api-access-ssmdc\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630435    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630445    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6mwr\" (UniqueName: \"kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-kube-api-access-v6mwr\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630453    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630461    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630469    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630476    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630484    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f881b2-0e66-4706-b9d1-4b1842b5d33c-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630493    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630501    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58017dd-1638-4df7-a7f9-dffd6db076f6-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630510    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630518    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58017dd-1638-4df7-a7f9-dffd6db076f6-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630526    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630536    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630581    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54eb3905-9d69-4cf9-8cc8-a279635e8183-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630590    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f881b2-0e66-4706-b9d1-4b1842b5d33c-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.630729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.630598    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vt7p\" (UniqueName: \"kubernetes.io/projected/e58017dd-1638-4df7-a7f9-dffd6db076f6-kube-api-access-9vt7p\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:16:32.710953 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:16:32.707066    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194c5fe5_da5e_4798_99c5_63a82e75ec1e.slice/crio-d37b741cd476de9e7ce194e10c31ee091e9172ac4bec9c7e2831f184a22e6747 WatchSource:0}: Error finding container d37b741cd476de9e7ce194e10c31ee091e9172ac4bec9c7e2831f184a22e6747: Status 404 returned error can't find the container with id d37b741cd476de9e7ce194e10c31ee091e9172ac4bec9c7e2831f184a22e6747
Oct 30 14:16:32.710953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.709917    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:16:32.710953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.709979    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:16:32.710953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.710026    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:16:32.713364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:32.713128    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"]
Oct 30 14:16:33.131857 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.131814    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" event={"ID":"194c5fe5-da5e-4798-99c5-63a82e75ec1e","Type":"ContainerStarted","Data":"70e6e394207ef5d567a269c071209769ff53bf8c3fb55718869e506f1d647f39"}
Oct 30 14:16:33.132307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.131887    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" event={"ID":"194c5fe5-da5e-4798-99c5-63a82e75ec1e","Type":"ContainerStarted","Data":"d37b741cd476de9e7ce194e10c31ee091e9172ac4bec9c7e2831f184a22e6747"}
Oct 30 14:16:33.134808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.134785    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"
Oct 30 14:16:33.134915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.134799    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79" event={"ID":"e58017dd-1638-4df7-a7f9-dffd6db076f6","Type":"ContainerDied","Data":"f4638efc3b354dcef9b753f505b7f0576c97cd8f2e76a41000c914d703d20739"}
Oct 30 14:16:33.134915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.134861    4530 scope.go:122] "RemoveContainer" containerID="69262667ecdf8d545bdb6167e6cb61256031486673cff2095bc82dedf9b9b17a"
Oct 30 14:16:33.137775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.137751    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md" event={"ID":"97052caf-e138-4d01-8b20-a8cade8b7a3d","Type":"ContainerDied","Data":"b200e1a95183ab33fb3eaa99ffc0a37c222a0be4fb762a4ee531c1f185c1ca37"}
Oct 30 14:16:33.137944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.137737    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"
Oct 30 14:16:33.140670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.140084    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c" event={"ID":"54eb3905-9d69-4cf9-8cc8-a279635e8183","Type":"ContainerDied","Data":"5a8d397cb672cec3e2a2becc2efe1e234341d74fb49bc15d659537b211dd2332"}
Oct 30 14:16:33.140670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.140189    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"
Oct 30 14:16:33.143082 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.143060    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4" event={"ID":"48f881b2-0e66-4706-b9d1-4b1842b5d33c","Type":"ContainerDied","Data":"0e4ba75b2cf75a06ff33ff87721c555e155a7bea9a53aaba8f6ea8ff0b4b6f7a"}
Oct 30 14:16:33.143186 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.143102    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"
Oct 30 14:16:33.153850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.153829    4530 scope.go:122] "RemoveContainer" containerID="9e1028d0d75b9cdb5ed3208f264c354558f5ca8ac8b8192e29e8f6f126a37f9f"
Oct 30 14:16:33.164967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.164928    4530 scope.go:122] "RemoveContainer" containerID="b8a69ae8796b70107b66408024a1ce4e2099ac2d2b6b14ff987fd763214f1a34"
Oct 30 14:16:33.178664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.178639    4530 scope.go:122] "RemoveContainer" containerID="4407ac55a8048fb70d2ac3e32039152b6b76d39aa8edb1f00deede39942054a9"
Oct 30 14:16:33.185876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.185820    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" podStartSLOduration=2.185803228 podStartE2EDuration="2.185803228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:16:33.182412928 +0000 UTC m=+1658.214311869" watchObservedRunningTime="2026-10-30 14:16:33.185803228 +0000 UTC m=+1658.217702168"
Oct 30 14:16:33.231990 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.231958    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"]
Oct 30 14:16:33.238923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.238895    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-qtq79"]
Oct 30 14:16:33.278901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.278867    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"]
Oct 30 14:16:33.284962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.284936    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-tr5m4"]
Oct 30 14:16:33.324324 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.324294    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"]
Oct 30 14:16:33.334894 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.334857    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7r7md"]
Oct 30 14:16:33.335015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.334911    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:33.336727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.336700    4530 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 14:16:33.336822 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.336763    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" podUID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 14:16:33.375134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.375108    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"]
Oct 30 14:16:33.383598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:33.383524    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-xlp9c"]
Oct 30 14:16:34.335087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:34.335050    4530 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 14:16:34.335520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:34.335123    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" podUID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 14:16:35.127014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:35.126979    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" path="/var/lib/kubelet/pods/48f881b2-0e66-4706-b9d1-4b1842b5d33c/volumes"
Oct 30 14:16:35.127714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:35.127695    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" path="/var/lib/kubelet/pods/54eb3905-9d69-4cf9-8cc8-a279635e8183/volumes"
Oct 30 14:16:35.128388 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:35.128371    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" path="/var/lib/kubelet/pods/97052caf-e138-4d01-8b20-a8cade8b7a3d/volumes"
Oct 30 14:16:35.129486 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:35.129465    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" path="/var/lib/kubelet/pods/e58017dd-1638-4df7-a7f9-dffd6db076f6/volumes"
Oct 30 14:16:35.335721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:35.335684    4530 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 14:16:35.336135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:35.335749    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" podUID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 14:16:36.335148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:36.335114    4530 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 14:16:36.335311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:36.335176    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" podUID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 14:16:37.355035 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:37.354996    4530 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 14:16:37.355737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:37.355704    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" podUID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 14:16:38.232903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:38.232862    4530 generic.go:356] "Generic (PLEG): container finished" podID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerID="3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0" exitCode=0
Oct 30 14:16:38.233118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:38.232943    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" event={"ID":"7ac20408-a58d-4bc9-8526-b86fe83d9020","Type":"ContainerDied","Data":"3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0"}
Oct 30 14:16:38.233443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:38.233429    4530 scope.go:122] "RemoveContainer" containerID="3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0"
Oct 30 14:16:38.235206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:38.235179    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:16:38.235283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:38.235246    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:16:38.235283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:38.235277    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:16:38.335958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:38.335925    4530 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 14:16:38.336148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:38.336016    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" podUID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 14:16:39.310156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:39.310073    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" event={"ID":"7ac20408-a58d-4bc9-8526-b86fe83d9020","Type":"ContainerStarted","Data":"79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab"}
Oct 30 14:16:39.345394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:39.343228    4530 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 14:16:39.345715 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:39.345681    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" podUID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 14:16:40.234536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:40.234500    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:16:40.234805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:40.234718    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 14:16:40.234805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:40.234762    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 14:16:40.339347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:40.339320    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:40.339762 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:40.339687    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:40.340354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:40.340336    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:16:41.235564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:41.234953    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 14:16:41.235564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:41.235039    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 14:16:41.426206 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:16:41.426170    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:16:42.039892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:42.039712    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"]
Oct 30 14:16:42.052178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:42.052150    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68f39a02-xlg8b"]
Oct 30 14:16:42.235089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:42.235042    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 14:16:42.235564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:42.235102    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 14:16:43.180221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:43.179812    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d" path="/var/lib/kubelet/pods/b74cb90f-b27c-49fd-b8eb-39dfaeaacb4d/volumes"
Oct 30 14:16:43.240564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:43.235020    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 14:16:43.240564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:43.235113    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 14:16:44.235228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:44.235179    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 14:16:44.235786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:44.235273    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 14:16:45.258841 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:45.258809    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:16:45.261784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:45.261760    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:16:45.265587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:45.265565    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:16:51.549037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:51.549001    4530 generic.go:356] "Generic (PLEG): container finished" podID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerID="eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4" exitCode=0
Oct 30 14:16:51.549489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:51.549082    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" event={"ID":"7bd9b019-951d-47cf-a812-946f4e10c8ba","Type":"ContainerDied","Data":"eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4"}
Oct 30 14:16:51.549646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:51.549624    4530 scope.go:122] "RemoveContainer" containerID="eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4"
Oct 30 14:16:51.551399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:51.551369    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:16:51.551461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:51.551443    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:16:51.551497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:51.551475    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:16:52.560008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:52.559975    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" event={"ID":"7bd9b019-951d-47cf-a812-946f4e10c8ba","Type":"ContainerStarted","Data":"061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b"}
Oct 30 14:16:52.801564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:52.790715    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:16:52.804608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:52.803703    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 14:16:52.804608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:52.803788    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 14:16:53.011776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:53.011683    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:16:53.215565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:53.208025    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"]
Oct 30 14:16:53.215565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:53.208077    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ae48173-2zqw2"]
Oct 30 14:16:53.808572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:53.808270    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 14:16:53.808572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:53.808345    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 14:16:54.796397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:54.795677    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 14:16:54.796397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:54.795775    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 14:16:55.257640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:55.255685    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea53d4d-91a5-44d3-9b42-e291de0856f7" path="/var/lib/kubelet/pods/0ea53d4d-91a5-44d3-9b42-e291de0856f7/volumes"
Oct 30 14:16:55.806954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:55.806914    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 14:16:55.807244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:55.807213    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 14:16:56.493755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:56.478785    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"]
Oct 30 14:16:56.493755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:56.479035    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy" containerID="cri-o://e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e" gracePeriod=30
Oct 30 14:16:56.760799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:56.759774    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"]
Oct 30 14:16:56.760799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:56.760044    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerName="istio-proxy" containerID="cri-o://3ec2a4ca4db28e38a56b9d336a3805c55c343615128f9eac27b87ded2d7ca17a" gracePeriod=30
Oct 30 14:16:56.847565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:56.840954    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 14:16:56.847565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:56.841030    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 14:16:56.952875 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:56.952833    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:16:57.053623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:57.053032    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"]
Oct 30 14:16:57.053623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:57.053285    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerName="istio-proxy" containerID="cri-o://74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184" gracePeriod=30
Oct 30 14:16:57.266459 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:57.266128    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"]
Oct 30 14:16:57.267037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:57.266999    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" containerName="istio-proxy" containerID="cri-o://54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec" gracePeriod=30
Oct 30 14:16:57.806809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:57.790304    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 14:16:58.014951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:58.014895    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 14:16:58.069620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:58.044846    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr" podUID="91d82203-dd8e-4e1f-ad76-0403aea6eaca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": context deadline exceeded"
Oct 30 14:16:58.069620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:58.057761    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 14:16:58.069620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:58.058072    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Oct 30 14:16:58.821731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:58.821673    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 14:16:58.861733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:58.861681    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 14:16:59.027280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:59.027229    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:16:59.049346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:59.049301    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr" podUID="91d82203-dd8e-4e1f-ad76-0403aea6eaca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:16:59.051149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:59.051123    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:16:59.065690 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:59.065643    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:16:59.082445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:59.082358    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:16:59.083961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:59.071892    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:16:59.085481 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:59.085449    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:16:59.108700 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:59.108651    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:16:59.851900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:59.851866    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:16:59.852570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:59.852530    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:16:59.893842 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:59.893789    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 14:16:59.894222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:16:59.894194    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 14:17:00.089068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:00.089034    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:17:00.328352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:00.301041    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:17:00.328352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:00.301435    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:17:00.330735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:00.330705    4530 scope.go:122] "RemoveContainer" containerID="97c5001a9b48bc14308076bdc284375e836957350c84d6a763738f7a28a3baf6"
Oct 30 14:17:00.432687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:00.421384    4530 scope.go:122] "RemoveContainer" containerID="62775a57688a2fd1a6c02e228254ab965072b215ce46b03ecd0f7ea9fa4fadc3"
Oct 30 14:17:00.458792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:00.448285    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:17:00.493969 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:00.493945    4530 scope.go:122] "RemoveContainer" containerID="fe2afd4906a470e8e6f3ffdc81ae66b793f60159e8b35ed3c961db2f9563e7c6"
Oct 30 14:17:00.539749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:00.534985    4530 scope.go:122] "RemoveContainer" containerID="56f222e6a5a9562bb5d77380bf80ca5fb2f9946637d61717ff76a1bf67910006"
Oct 30 14:17:00.567122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:00.567091    4530 scope.go:122] "RemoveContainer" containerID="e7432b6dc762377ff4567d967bbd2680aa8e9e6759f7a8c57f3590f809effcae"
Oct 30 14:17:00.581600 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:00.581484    4530 scope.go:122] "RemoveContainer" containerID="2756000688f45d5fb81e28201007da27f0f6cc6eaabb1f6694bbe2664cd21aea"
Oct 30 14:17:00.910945 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:00.910900    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:17:00.912703 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:00.912681    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:17:01.639770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.639740    4530 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:17:01.639896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.639798    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:17:01.844656 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.841775    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:17:01.930403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.930367    4530 generic.go:356] "Generic (PLEG): container finished" podID="849e6253-e63b-4f69-a345-15cef381ae4c" containerID="e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e" exitCode=0
Oct 30 14:17:01.930889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.930418    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"
Oct 30 14:17:01.930889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.930449    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" event={"ID":"849e6253-e63b-4f69-a345-15cef381ae4c","Type":"ContainerDied","Data":"e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e"}
Oct 30 14:17:01.930889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.930500    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k" event={"ID":"849e6253-e63b-4f69-a345-15cef381ae4c","Type":"ContainerDied","Data":"1a3ea61cba12dc7dcb5b506ed8c1cb6a103ae04782ce8393c80aeee09154e611"}
Oct 30 14:17:01.930889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.930531    4530 scope.go:122] "RemoveContainer" containerID="e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e"
Oct 30 14:17:01.932860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.932831    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" event={"ID":"acd5bb1a-fceb-4ca4-997a-e197ea4277c1","Type":"ContainerDied","Data":"3ec2a4ca4db28e38a56b9d336a3805c55c343615128f9eac27b87ded2d7ca17a"}
Oct 30 14:17:01.933003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.932840    4530 generic.go:356] "Generic (PLEG): container finished" podID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerID="3ec2a4ca4db28e38a56b9d336a3805c55c343615128f9eac27b87ded2d7ca17a" exitCode=0
Oct 30 14:17:01.942671 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.942642    4530 scope.go:122] "RemoveContainer" containerID="e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e"
Oct 30 14:17:01.942963 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:01.942939    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e\": container with ID starting with e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e not found: ID does not exist" containerID="e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e"
Oct 30 14:17:01.943041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.942976    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e"} err="failed to get container status \"e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e\": rpc error: code = NotFound desc = could not find container \"e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e\": container with ID starting with e42d550523dcfcd47fce928080ba569a795611dabc46b0bd26e6089c2077516e not found: ID does not exist"
Oct 30 14:17:01.961653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.961629    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-kube-api-access-7rgjt\" (UniqueName: \"kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-kube-api-access-7rgjt\") pod \"849e6253-e63b-4f69-a345-15cef381ae4c\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") "
Oct 30 14:17:01.961783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.961682    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-certs\") pod \"849e6253-e63b-4f69-a345-15cef381ae4c\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") "
Oct 30 14:17:01.961783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.961713    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-envoy\") pod \"849e6253-e63b-4f69-a345-15cef381ae4c\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") "
Oct 30 14:17:01.961901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.961830    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/849e6253-e63b-4f69-a345-15cef381ae4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849e6253-e63b-4f69-a345-15cef381ae4c-istiod-ca-cert\") pod \"849e6253-e63b-4f69-a345-15cef381ae4c\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") "
Oct 30 14:17:01.961901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.961874    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-credential-socket\") pod \"849e6253-e63b-4f69-a345-15cef381ae4c\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") "
Oct 30 14:17:01.962012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.961901    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-data\") pod \"849e6253-e63b-4f69-a345-15cef381ae4c\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") "
Oct 30 14:17:01.962069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962011    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-certs" pod "849e6253-e63b-4f69-a345-15cef381ae4c" (UID: "849e6253-e63b-4f69-a345-15cef381ae4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:01.962069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962022    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-socket\") pod \"849e6253-e63b-4f69-a345-15cef381ae4c\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") "
Oct 30 14:17:01.962234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962214    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/849e6253-e63b-4f69-a345-15cef381ae4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849e6253-e63b-4f69-a345-15cef381ae4c-istio-podinfo\") pod \"849e6253-e63b-4f69-a345-15cef381ae4c\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") "
Oct 30 14:17:01.962319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962248    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-socket" pod "849e6253-e63b-4f69-a345-15cef381ae4c" (UID: "849e6253-e63b-4f69-a345-15cef381ae4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:01.962319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962258    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-istio-token\" (UniqueName: \"kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-istio-token\") pod \"849e6253-e63b-4f69-a345-15cef381ae4c\" (UID: \"849e6253-e63b-4f69-a345-15cef381ae4c\") "
Oct 30 14:17:01.962319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962261    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-data" pod "849e6253-e63b-4f69-a345-15cef381ae4c" (UID: "849e6253-e63b-4f69-a345-15cef381ae4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:01.962319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962315    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-credential-socket" pod "849e6253-e63b-4f69-a345-15cef381ae4c" (UID: "849e6253-e63b-4f69-a345-15cef381ae4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:01.962530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962314    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849e6253-e63b-4f69-a345-15cef381ae4c-istiod-ca-cert" pod "849e6253-e63b-4f69-a345-15cef381ae4c" (UID: "849e6253-e63b-4f69-a345-15cef381ae4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:17:01.962940 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962921    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:01.962940 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962941    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:01.963089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962951    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849e6253-e63b-4f69-a345-15cef381ae4c-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:01.963089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962959    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:01.963089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.962972    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:01.964106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.964084    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-envoy" pod "849e6253-e63b-4f69-a345-15cef381ae4c" (UID: "849e6253-e63b-4f69-a345-15cef381ae4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:01.964356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.964335    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/849e6253-e63b-4f69-a345-15cef381ae4c-istio-podinfo" pod "849e6253-e63b-4f69-a345-15cef381ae4c" (UID: "849e6253-e63b-4f69-a345-15cef381ae4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:17:01.964446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.964427    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-kube-api-access-7rgjt" pod "849e6253-e63b-4f69-a345-15cef381ae4c" (UID: "849e6253-e63b-4f69-a345-15cef381ae4c"). InnerVolumeSpecName "kube-api-access-7rgjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:17:01.964590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:01.964534    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-istio-token" pod "849e6253-e63b-4f69-a345-15cef381ae4c" (UID: "849e6253-e63b-4f69-a345-15cef381ae4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:17:02.064858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.064820    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849e6253-e63b-4f69-a345-15cef381ae4c-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.064858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.064858    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849e6253-e63b-4f69-a345-15cef381ae4c-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.065082 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.065038    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.065082 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.065057    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rgjt\" (UniqueName: \"kubernetes.io/projected/849e6253-e63b-4f69-a345-15cef381ae4c-kube-api-access-7rgjt\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.342670    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9591-a48680"]
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344403    4530 cpu_manager.go:414] "containerMap: removing container" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344428    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344458    4530 cpu_manager.go:414] "containerMap: removing container" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344465    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344488    4530 cpu_manager.go:414] "containerMap: removing container" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344496    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344506    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344514    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344535    4530 cpu_manager.go:414] "containerMap: removing container" podUID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344559    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344580    4530 cpu_manager.go:414] "containerMap: removing container" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344586    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344763    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344780    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344792    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="54eb3905-9d69-4cf9-8cc8-a279635e8183" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344802    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58017dd-1638-4df7-a7f9-dffd6db076f6" containerName="istio-proxy"
Oct 30 14:17:02.345607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.344814    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f881b2-0e66-4706-b9d1-4b1842b5d33c" containerName="istio-proxy"
Oct 30 14:17:02.351703 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.351679    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"]
Oct 30 14:17:02.351970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.351832    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-a48680"
Oct 30 14:17:02.359058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.359032    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-6ck5k"]
Oct 30 14:17:02.364679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.364654    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9591-a48680"]
Oct 30 14:17:02.430489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.430459    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:17:02.473010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.472971    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-credential-socket\") pod \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") "
Oct 30 14:17:02.473213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.473050    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-kube-api-access-fztdf\" (UniqueName: \"kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-kube-api-access-fztdf\") pod \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") "
Oct 30 14:17:02.473213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.473185    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-token\" (UniqueName: \"kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-token\") pod \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") "
Oct 30 14:17:02.473354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.473313    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-envoy\") pod \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") "
Oct 30 14:17:02.473413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.473350    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-socket\") pod \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") "
Oct 30 14:17:02.473473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.473423    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-data\") pod \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") "
Oct 30 14:17:02.473530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.473487    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-podinfo\") pod \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") "
Oct 30 14:17:02.473650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.473630    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-certs\") pod \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") "
Oct 30 14:17:02.473734 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.473703    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istiod-ca-cert\") pod \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\" (UID: \"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7\") "
Oct 30 14:17:02.474510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.474482    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnslk\" (UniqueName: \"kubernetes.io/projected/8907340d-1149-4ffc-984f-3fcd03e1683a-kube-api-access-rnslk\") pod \"test-curl-9591-a48680\" (UID: \"8907340d-1149-4ffc-984f-3fcd03e1683a\") " pod="opendatahub/test-curl-9591-a48680"
Oct 30 14:17:02.474992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.474966    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-credential-socket" pod "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" (UID: "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.475587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.475552    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-data" pod "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" (UID: "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.476602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.476311    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-socket" pod "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" (UID: "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.476767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.476715    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istiod-ca-cert" pod "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" (UID: "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:17:02.477021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.476964    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-certs" pod "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" (UID: "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.503565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.501968    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-kube-api-access-fztdf" pod "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" (UID: "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7"). InnerVolumeSpecName "kube-api-access-fztdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:17:02.504328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.504273    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-podinfo" pod "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" (UID: "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:17:02.510806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.510770    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-token" pod "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" (UID: "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:17:02.513767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.513673    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-envoy" pod "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" (UID: "eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.576951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.576374    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnslk\" (UniqueName: \"kubernetes.io/projected/8907340d-1149-4ffc-984f-3fcd03e1683a-kube-api-access-rnslk\") pod \"test-curl-9591-a48680\" (UID: \"8907340d-1149-4ffc-984f-3fcd03e1683a\") " pod="opendatahub/test-curl-9591-a48680"
Oct 30 14:17:02.576951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.576699    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.576951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.576721    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fztdf\" (UniqueName: \"kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-kube-api-access-fztdf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.576951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.576736    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.576951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.576748    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.576951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.576761    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.576951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.576773    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.576951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.576787    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.576951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.576801    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.576951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.576814    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.634661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.634625    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnslk\" (UniqueName: \"kubernetes.io/projected/8907340d-1149-4ffc-984f-3fcd03e1683a-kube-api-access-rnslk\") pod \"test-curl-9591-a48680\" (UID: \"8907340d-1149-4ffc-984f-3fcd03e1683a\") " pod="opendatahub/test-curl-9591-a48680"
Oct 30 14:17:02.678705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.678679    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:17:02.707569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.707526    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:17:02.729962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.729936    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-a48680"
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.817828    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4156f6-7078-48ed-8a52-9fc04b755118-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4156f6-7078-48ed-8a52-9fc04b755118-istiod-ca-cert\") pod \"da4156f6-7078-48ed-8a52-9fc04b755118\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.817879    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-token\" (UniqueName: \"kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-token\") pod \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.817930    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-credential-socket\") pod \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818231    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-credential-socket" pod "acd5bb1a-fceb-4ca4-997a-e197ea4277c1" (UID: "acd5bb1a-fceb-4ca4-997a-e197ea4277c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818233    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4156f6-7078-48ed-8a52-9fc04b755118-istiod-ca-cert" pod "da4156f6-7078-48ed-8a52-9fc04b755118" (UID: "da4156f6-7078-48ed-8a52-9fc04b755118"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818262    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-envoy\") pod \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818351    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-certs\") pod \"da4156f6-7078-48ed-8a52-9fc04b755118\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818377    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-data\") pod \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818438    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-socket\") pod \"da4156f6-7078-48ed-8a52-9fc04b755118\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818457    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da4156f6-7078-48ed-8a52-9fc04b755118-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4156f6-7078-48ed-8a52-9fc04b755118-istio-podinfo\") pod \"da4156f6-7078-48ed-8a52-9fc04b755118\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818483    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-socket\") pod \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818510    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-podinfo\") pod \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818530    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-istio-token\" (UniqueName: \"kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-istio-token\") pod \"da4156f6-7078-48ed-8a52-9fc04b755118\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818697    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-envoy\") pod \"da4156f6-7078-48ed-8a52-9fc04b755118\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818729    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istiod-ca-cert\") pod \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818773    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-certs\") pod \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818817    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-kube-api-access-xmmbb\" (UniqueName: \"kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-kube-api-access-xmmbb\") pod \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\" (UID: \"acd5bb1a-fceb-4ca4-997a-e197ea4277c1\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818876    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-data\") pod \"da4156f6-7078-48ed-8a52-9fc04b755118\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.818926    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-credential-socket\") pod \"da4156f6-7078-48ed-8a52-9fc04b755118\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") "
Oct 30 14:17:02.819801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.819012    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-kube-api-access-gf59q\" (UniqueName: \"kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-kube-api-access-gf59q\") pod \"da4156f6-7078-48ed-8a52-9fc04b755118\" (UID: \"da4156f6-7078-48ed-8a52-9fc04b755118\") "
Oct 30 14:17:02.820791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.819941    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4156f6-7078-48ed-8a52-9fc04b755118-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.820791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.819969    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.827559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.827513    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-certs" pod "acd5bb1a-fceb-4ca4-997a-e197ea4277c1" (UID: "acd5bb1a-fceb-4ca4-997a-e197ea4277c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.831869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.831520    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-data" pod "da4156f6-7078-48ed-8a52-9fc04b755118" (UID: "da4156f6-7078-48ed-8a52-9fc04b755118"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.835375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.835073    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istiod-ca-cert" pod "acd5bb1a-fceb-4ca4-997a-e197ea4277c1" (UID: "acd5bb1a-fceb-4ca4-997a-e197ea4277c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:17:02.835795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.835572    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-data" pod "acd5bb1a-fceb-4ca4-997a-e197ea4277c1" (UID: "acd5bb1a-fceb-4ca4-997a-e197ea4277c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.843068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.843035    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-certs" pod "da4156f6-7078-48ed-8a52-9fc04b755118" (UID: "da4156f6-7078-48ed-8a52-9fc04b755118"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.854675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.845135    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-socket" pod "da4156f6-7078-48ed-8a52-9fc04b755118" (UID: "da4156f6-7078-48ed-8a52-9fc04b755118"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.854675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.845979    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-envoy" pod "acd5bb1a-fceb-4ca4-997a-e197ea4277c1" (UID: "acd5bb1a-fceb-4ca4-997a-e197ea4277c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.854675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.847308    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-token" pod "acd5bb1a-fceb-4ca4-997a-e197ea4277c1" (UID: "acd5bb1a-fceb-4ca4-997a-e197ea4277c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:17:02.854675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.848124    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-credential-socket" pod "da4156f6-7078-48ed-8a52-9fc04b755118" (UID: "da4156f6-7078-48ed-8a52-9fc04b755118"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.854675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.848691    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-podinfo" pod "acd5bb1a-fceb-4ca4-997a-e197ea4277c1" (UID: "acd5bb1a-fceb-4ca4-997a-e197ea4277c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:17:02.854675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.848966    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-kube-api-access-gf59q" pod "da4156f6-7078-48ed-8a52-9fc04b755118" (UID: "da4156f6-7078-48ed-8a52-9fc04b755118"). InnerVolumeSpecName "kube-api-access-gf59q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:17:02.854675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.851654    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-socket" pod "acd5bb1a-fceb-4ca4-997a-e197ea4277c1" (UID: "acd5bb1a-fceb-4ca4-997a-e197ea4277c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.855233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.855206    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-envoy" pod "da4156f6-7078-48ed-8a52-9fc04b755118" (UID: "da4156f6-7078-48ed-8a52-9fc04b755118"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:02.855784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.855758    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da4156f6-7078-48ed-8a52-9fc04b755118-istio-podinfo" pod "da4156f6-7078-48ed-8a52-9fc04b755118" (UID: "da4156f6-7078-48ed-8a52-9fc04b755118"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:17:02.856779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.856748    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-kube-api-access-xmmbb" pod "acd5bb1a-fceb-4ca4-997a-e197ea4277c1" (UID: "acd5bb1a-fceb-4ca4-997a-e197ea4277c1"). InnerVolumeSpecName "kube-api-access-xmmbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:17:02.856902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.856877    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-istio-token" pod "da4156f6-7078-48ed-8a52-9fc04b755118" (UID: "da4156f6-7078-48ed-8a52-9fc04b755118"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:17:02.923007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.922914    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.923222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.923202    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.923369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.923346    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.923509 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.923492    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4156f6-7078-48ed-8a52-9fc04b755118-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.923637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.923622    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.923723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.923711    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.923965 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.923948    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.924072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.924060    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.924421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.924166    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.924591 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.924575    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.924710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.924692    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmmbb\" (UniqueName: \"kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-kube-api-access-xmmbb\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.924807 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.924793    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.924913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.924900    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4156f6-7078-48ed-8a52-9fc04b755118-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.925006 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.924994    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf59q\" (UniqueName: \"kubernetes.io/projected/da4156f6-7078-48ed-8a52-9fc04b755118-kube-api-access-gf59q\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.925105 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.925092    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.925189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.925178    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd5bb1a-fceb-4ca4-997a-e197ea4277c1-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:02.951685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.951611    4530 generic.go:356] "Generic (PLEG): container finished" podID="da4156f6-7078-48ed-8a52-9fc04b755118" containerID="54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec" exitCode=0
Oct 30 14:17:02.951685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.951658    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"
Oct 30 14:17:02.952882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.951656    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" event={"ID":"da4156f6-7078-48ed-8a52-9fc04b755118","Type":"ContainerDied","Data":"54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec"}
Oct 30 14:17:02.952882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.951877    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw" event={"ID":"da4156f6-7078-48ed-8a52-9fc04b755118","Type":"ContainerDied","Data":"5af30b6fc480fc2ad01c8c0b79cad5fcf97398fecac02369246d9d042e33f72e"}
Oct 30 14:17:02.952882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.951917    4530 scope.go:122] "RemoveContainer" containerID="54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec"
Oct 30 14:17:02.955051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.955026    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"
Oct 30 14:17:02.955582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.955559    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv" event={"ID":"acd5bb1a-fceb-4ca4-997a-e197ea4277c1","Type":"ContainerDied","Data":"31af9344ccb8a00bc5076d31e9253055790e89520ca101c48d1dc1406904739c"}
Oct 30 14:17:02.956863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.956835    4530 generic.go:356] "Generic (PLEG): container finished" podID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerID="74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184" exitCode=0
Oct 30 14:17:02.956978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.956909    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" event={"ID":"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7","Type":"ContainerDied","Data":"74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184"}
Oct 30 14:17:02.956978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.956949    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl" event={"ID":"eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7","Type":"ContainerDied","Data":"f27b940ff062a0b7bc82794e69bbae2353ac6a97a077a8b7c6de01311a059cb2"}
Oct 30 14:17:02.957090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.957019    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"
Oct 30 14:17:02.971355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.971333    4530 scope.go:122] "RemoveContainer" containerID="54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec"
Oct 30 14:17:02.971736 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:02.971708    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec\": container with ID starting with 54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec not found: ID does not exist" containerID="54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec"
Oct 30 14:17:02.971823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.971750    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec"} err="failed to get container status \"54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec\": rpc error: code = NotFound desc = could not find container \"54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec\": container with ID starting with 54d3848145f8ae5773c038fb320f31ead61c8a44dec517df1c51c8c2399d84ec not found: ID does not exist"
Oct 30 14:17:02.971823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:02.971772    4530 scope.go:122] "RemoveContainer" containerID="3ec2a4ca4db28e38a56b9d336a3805c55c343615128f9eac27b87ded2d7ca17a"
Oct 30 14:17:03.011606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.011574    4530 scope.go:122] "RemoveContainer" containerID="74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184"
Oct 30 14:17:03.024317 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.024295    4530 scope.go:122] "RemoveContainer" containerID="74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184"
Oct 30 14:17:03.024689 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:03.024653    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184\": container with ID starting with 74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184 not found: ID does not exist" containerID="74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184"
Oct 30 14:17:03.024778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.024696    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184"} err="failed to get container status \"74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184\": rpc error: code = NotFound desc = could not find container \"74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184\": container with ID starting with 74db44d4cae90e9614d9a134fdf9a58649229b4e73703e6a9b49fb424fb26184 not found: ID does not exist"
Oct 30 14:17:03.219562 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:17:03.218724    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8907340d_1149_4ffc_984f_3fcd03e1683a.slice/crio-bbe63a30da06367cf57ed16c979dc297b2ab07b0d6182f49a7ca0a9307aa1b1c WatchSource:0}: Error finding container bbe63a30da06367cf57ed16c979dc297b2ab07b0d6182f49a7ca0a9307aa1b1c: Status 404 returned error can't find the container with id bbe63a30da06367cf57ed16c979dc297b2ab07b0d6182f49a7ca0a9307aa1b1c
Oct 30 14:17:03.254235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.254203    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="849e6253-e63b-4f69-a345-15cef381ae4c" path="/var/lib/kubelet/pods/849e6253-e63b-4f69-a345-15cef381ae4c/volumes"
Oct 30 14:17:03.254998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.254976    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"]
Oct 30 14:17:03.255169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.255150    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-r4qqv"]
Oct 30 14:17:03.255261 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.255188    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9591-a48680"]
Oct 30 14:17:03.255261 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.255212    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"]
Oct 30 14:17:03.258424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.258398    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-nbnsl"]
Oct 30 14:17:03.270637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.270611    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"]
Oct 30 14:17:03.275828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.275687    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-xnfpw"]
Oct 30 14:17:03.970271 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.970182    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9591-a48680" event={"ID":"8907340d-1149-4ffc-984f-3fcd03e1683a","Type":"ContainerStarted","Data":"027a54769bb89ec4c9b742133771edf181ee4ff490e28252a9d6a11c494a7be5"}
Oct 30 14:17:03.970271 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:03.970222    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9591-a48680" event={"ID":"8907340d-1149-4ffc-984f-3fcd03e1683a","Type":"ContainerStarted","Data":"bbe63a30da06367cf57ed16c979dc297b2ab07b0d6182f49a7ca0a9307aa1b1c"}
Oct 30 14:17:04.028304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:04.028252    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9591-a48680" podStartSLOduration=2.028238683 podStartE2EDuration="2.028238683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:04.02701135 +0000 UTC m=+1689.058910290" watchObservedRunningTime="2026-10-30 14:17:04.028238683 +0000 UTC m=+1689.060137621"
Oct 30 14:17:05.009254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.009219    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9591-a48680"]
Oct 30 14:17:05.015275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.015243    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9591-a48680"]
Oct 30 14:17:05.249021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.248985    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" path="/var/lib/kubelet/pods/acd5bb1a-fceb-4ca4-997a-e197ea4277c1/volumes"
Oct 30 14:17:05.249600 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.249583    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" path="/var/lib/kubelet/pods/da4156f6-7078-48ed-8a52-9fc04b755118/volumes"
Oct 30 14:17:05.250086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.250072    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" path="/var/lib/kubelet/pods/eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7/volumes"
Oct 30 14:17:05.912493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.912455    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"]
Oct 30 14:17:05.913885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.913858    4530 cpu_manager.go:414] "containerMap: removing container" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" containerName="istio-proxy"
Oct 30 14:17:05.914027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.914013    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" containerName="istio-proxy"
Oct 30 14:17:05.914138 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.914127    4530 cpu_manager.go:414] "containerMap: removing container" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerName="istio-proxy"
Oct 30 14:17:05.914240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.914218    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerName="istio-proxy"
Oct 30 14:17:05.914355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.914261    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8907340d-1149-4ffc-984f-3fcd03e1683a" containerName="test-curl-9591-a48680"
Oct 30 14:17:05.914355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.914270    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8907340d-1149-4ffc-984f-3fcd03e1683a" containerName="test-curl-9591-a48680"
Oct 30 14:17:05.914355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.914310    4530 cpu_manager.go:414] "containerMap: removing container" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerName="istio-proxy"
Oct 30 14:17:05.914355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.914319    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerName="istio-proxy"
Oct 30 14:17:05.914595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.914520    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="97052caf-e138-4d01-8b20-a8cade8b7a3d" containerName="istio-proxy"
Oct 30 14:17:05.914595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.914534    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8907340d-1149-4ffc-984f-3fcd03e1683a" containerName="test-curl-9591-a48680"
Oct 30 14:17:05.914595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.914568    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef3270b-eec9-4ca5-9b4d-d6faf51ce1d7" containerName="istio-proxy"
Oct 30 14:17:05.914595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.914583    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4156f6-7078-48ed-8a52-9fc04b755118" containerName="istio-proxy"
Oct 30 14:17:05.914595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.914593    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd5bb1a-fceb-4ca4-997a-e197ea4277c1" containerName="istio-proxy"
Oct 30 14:17:05.919710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.919688    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:05.946243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.946208    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-972fee-openshift-default-dockercfg-9jlpv\""
Oct 30 14:17:05.947061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.947036    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"]
Oct 30 14:17:05.993966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:05.993900    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9591-a48680" podUID="8907340d-1149-4ffc-984f-3fcd03e1683a" containerName="test-curl-9591-a48680" containerID="cri-o://027a54769bb89ec4c9b742133771edf181ee4ff490e28252a9d6a11c494a7be5" gracePeriod=2
Oct 30 14:17:06.138574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.117508    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-certs\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.138574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.117591    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istiod-ca-cert\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.138574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.117672    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-socket\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.138574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.117728    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-credential-socket\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.138574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.117850    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9n7p\" (UniqueName: \"kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-kube-api-access-t9n7p\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.138574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.117891    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-token\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.138574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.117930    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-data\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.138574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.117993    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-envoy\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.138574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.118038    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-podinfo\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.217705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.216686    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"]
Oct 30 14:17:06.245561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.230166    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-envoy\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.245561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.230233    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-podinfo\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.245561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.230349    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-certs\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.245561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.230411    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istiod-ca-cert\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.245561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.230524    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-socket\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.245561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.230604    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-credential-socket\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.245561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.230780    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9n7p\" (UniqueName: \"kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-kube-api-access-t9n7p\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.245561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.230830    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-token\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.245561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.230867    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-data\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.245561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.231396    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-data\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.246521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.246337    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-socket\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.246683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.246659    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-credential-socket\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.279593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.265395    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-certs\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.279593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.266264    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istiod-ca-cert\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.322123 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.322056    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9n7p\" (UniqueName: \"kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-kube-api-access-t9n7p\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.350727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.346779    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eaeaa01-bsrvh"]
Oct 30 14:17:06.374580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.369970    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-podinfo\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.374580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.370320    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-envoy\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.393558 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.390903    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-token\") pod \"e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") " pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.548591 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.548564    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:06.880197 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:17:06.880166    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3e9eea_3660_4b16_8cc8_afe897b3b3b2.slice/crio-a775cf788670a4d46e412e971f5e02820f0178ffac184fcac45a3fcabf2912e5 WatchSource:0}: Error finding container a775cf788670a4d46e412e971f5e02820f0178ffac184fcac45a3fcabf2912e5: Status 404 returned error can't find the container with id a775cf788670a4d46e412e971f5e02820f0178ffac184fcac45a3fcabf2912e5
Oct 30 14:17:06.882984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.882943    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:06.883101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.883032    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:06.883101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.883078    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:06.884264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:06.884238    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"]
Oct 30 14:17:07.061864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:07.061806    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" event={"ID":"af3e9eea-3660-4b16-8cc8-afe897b3b3b2","Type":"ContainerStarted","Data":"a775cf788670a4d46e412e971f5e02820f0178ffac184fcac45a3fcabf2912e5"}
Oct 30 14:17:07.120123 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:07.119959    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d372b404-a53e-4865-88a3-b6cba5a3f5f8" path="/var/lib/kubelet/pods/d372b404-a53e-4865-88a3-b6cba5a3f5f8/volumes"
Oct 30 14:17:08.074103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:08.074069    4530 generic.go:356] "Generic (PLEG): container finished" podID="8907340d-1149-4ffc-984f-3fcd03e1683a" containerID="027a54769bb89ec4c9b742133771edf181ee4ff490e28252a9d6a11c494a7be5" exitCode=137
Oct 30 14:17:08.076581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:08.076550    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" event={"ID":"af3e9eea-3660-4b16-8cc8-afe897b3b3b2","Type":"ContainerStarted","Data":"b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b"}
Oct 30 14:17:08.159010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:08.158959    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" podStartSLOduration=3.158943117 podStartE2EDuration="3.158943117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:08.144401692 +0000 UTC m=+1693.176300631" watchObservedRunningTime="2026-10-30 14:17:08.158943117 +0000 UTC m=+1693.190842056"
Oct 30 14:17:08.224128 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:08.224099    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-a48680"
Oct 30 14:17:08.312107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:08.312068    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8907340d-1149-4ffc-984f-3fcd03e1683a-kube-api-access-rnslk\" (UniqueName: \"kubernetes.io/projected/8907340d-1149-4ffc-984f-3fcd03e1683a-kube-api-access-rnslk\") pod \"8907340d-1149-4ffc-984f-3fcd03e1683a\" (UID: \"8907340d-1149-4ffc-984f-3fcd03e1683a\") "
Oct 30 14:17:08.334997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:08.319984    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8907340d-1149-4ffc-984f-3fcd03e1683a-kube-api-access-rnslk" pod "8907340d-1149-4ffc-984f-3fcd03e1683a" (UID: "8907340d-1149-4ffc-984f-3fcd03e1683a"). InnerVolumeSpecName "kube-api-access-rnslk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:17:08.437605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:08.432988    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnslk\" (UniqueName: \"kubernetes.io/projected/8907340d-1149-4ffc-984f-3fcd03e1683a-kube-api-access-rnslk\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:08.560891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:08.559121    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:08.577367 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:08.576947    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:09.088241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:09.088209    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-a48680"
Oct 30 14:17:09.088241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:09.088241    4530 scope.go:122] "RemoveContainer" containerID="027a54769bb89ec4c9b742133771edf181ee4ff490e28252a9d6a11c494a7be5"
Oct 30 14:17:09.089045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:09.089023    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:09.090200 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:09.090179    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:17:09.117784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:09.117753    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8907340d-1149-4ffc-984f-3fcd03e1683a" path="/var/lib/kubelet/pods/8907340d-1149-4ffc-984f-3fcd03e1683a/volumes"
Oct 30 14:17:09.598456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:09.598415    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9603-4db048"]
Oct 30 14:17:09.598701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:09.598674    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9603-4db048" podUID="e56997e4-1701-4fd2-8ddb-b12da63a113e" containerName="test-curl-9603-4db048" containerID="cri-o://60cf8cb5c00ddf9ca17fdac8b288e67fc04652b7d254b32decb3a8dff63c7100" gracePeriod=2
Oct 30 14:17:09.607148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:09.607116    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9603-4db048"]
Oct 30 14:17:09.614837 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:09.614804    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-4db048\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e56997e4-1701-4fd2-8ddb-b12da63a113e" pod="opendatahub/test-curl-9603-4db048"
Oct 30 14:17:10.586154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.586116    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"]
Oct 30 14:17:10.587204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.587182    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e56997e4-1701-4fd2-8ddb-b12da63a113e" containerName="test-curl-9603-4db048"
Oct 30 14:17:10.587204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.587203    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56997e4-1701-4fd2-8ddb-b12da63a113e" containerName="test-curl-9603-4db048"
Oct 30 14:17:10.587401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.587349    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56997e4-1701-4fd2-8ddb-b12da63a113e" containerName="test-curl-9603-4db048"
Oct 30 14:17:10.590854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.590832    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.602370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.602336    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-socket\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.602668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.602645    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-certs\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.603179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.603157    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-data\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.603373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.603335    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8631d1a-7032-493f-be3d-131ac728904a-istiod-ca-cert\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.603497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.603401    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-envoy\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.603497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.603491    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8631d1a-7032-493f-be3d-131ac728904a-istio-podinfo\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.603715 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.603566    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-istio-token\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.603715 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.603657    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz59l\" (UniqueName: \"kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-kube-api-access-rz59l\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.603715 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.603687    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-credential-socket\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.621208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.621177    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"]
Oct 30 14:17:10.626248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.626226    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d727dc-openshift-default-dockercfg-mkbw7\""
Oct 30 14:17:10.658800 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:10.658759    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-4db048\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e56997e4-1701-4fd2-8ddb-b12da63a113e" pod="opendatahub/test-curl-9603-4db048"
Oct 30 14:17:10.704599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.704563    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-istio-token\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.704770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.704629    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz59l\" (UniqueName: \"kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-kube-api-access-rz59l\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.704770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.704650    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-credential-socket\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.704770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.704719    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-socket\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.704770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.704749    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-certs\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.704969 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.704809    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-data\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.704969 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.704856    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8631d1a-7032-493f-be3d-131ac728904a-istiod-ca-cert\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.704969 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.704896    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-envoy\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.705085 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.704973    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8631d1a-7032-493f-be3d-131ac728904a-istio-podinfo\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.705172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.705133    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-credential-socket\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.705220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.705165    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-certs\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.705279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.705221    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-socket\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.705425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.705401    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-data\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.705512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.705496    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8631d1a-7032-493f-be3d-131ac728904a-istiod-ca-cert\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.707397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.707366    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-envoy\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.707586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.707561    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8631d1a-7032-493f-be3d-131ac728904a-istio-podinfo\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.731497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.731410    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-istio-token\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.731663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.731513    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz59l\" (UniqueName: \"kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-kube-api-access-rz59l\") pod \"e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") " pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:10.914616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:10.914253    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:11.170631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.170508    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"]
Oct 30 14:17:11.205031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.204980    4530 generic.go:356] "Generic (PLEG): container finished" podID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerID="e475cb9e369d7116ad6ddac404d7a1a5ca626df82a91b67afe04079a0d2fd7ae" exitCode=0
Oct 30 14:17:11.210645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.207121    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" event={"ID":"42dedd41-345f-4dd6-8519-fa7587c4a383","Type":"ContainerDied","Data":"e475cb9e369d7116ad6ddac404d7a1a5ca626df82a91b67afe04079a0d2fd7ae"}
Oct 30 14:17:11.219376 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.218294    4530 scope.go:122] "RemoveContainer" containerID="e475cb9e369d7116ad6ddac404d7a1a5ca626df82a91b67afe04079a0d2fd7ae"
Oct 30 14:17:11.228346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.228308    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:11.228699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.228602    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:11.228914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.228890    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:11.238913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.236418    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06755c7-4zgjk"]
Oct 30 14:17:11.395201 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:11.392320    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:17:11.558708 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:17:11.558672    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8631d1a_7032_493f_be3d_131ac728904a.slice/crio-b6bca2e6b0cc241c6f6276500125b6c3d855dc0edf3e94b6f8166298b7c2655d WatchSource:0}: Error finding container b6bca2e6b0cc241c6f6276500125b6c3d855dc0edf3e94b6f8166298b7c2655d: Status 404 returned error can't find the container with id b6bca2e6b0cc241c6f6276500125b6c3d855dc0edf3e94b6f8166298b7c2655d
Oct 30 14:17:11.560707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.560587    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"]
Oct 30 14:17:11.562056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.562022    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:11.562153 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.562090    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:11.562153 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.562118    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:11.840165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.840120    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-4db048"
Oct 30 14:17:11.974351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.974318    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56997e4-1701-4fd2-8ddb-b12da63a113e-kube-api-access-hx666\" (UniqueName: \"kubernetes.io/projected/e56997e4-1701-4fd2-8ddb-b12da63a113e-kube-api-access-hx666\") pod \"e56997e4-1701-4fd2-8ddb-b12da63a113e\" (UID: \"e56997e4-1701-4fd2-8ddb-b12da63a113e\") "
Oct 30 14:17:11.976477 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:11.976447    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56997e4-1701-4fd2-8ddb-b12da63a113e-kube-api-access-hx666" pod "e56997e4-1701-4fd2-8ddb-b12da63a113e" (UID: "e56997e4-1701-4fd2-8ddb-b12da63a113e"). InnerVolumeSpecName "kube-api-access-hx666". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:17:12.076706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.076668    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx666\" (UniqueName: \"kubernetes.io/projected/e56997e4-1701-4fd2-8ddb-b12da63a113e-kube-api-access-hx666\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:12.215784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.215744    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" event={"ID":"42dedd41-345f-4dd6-8519-fa7587c4a383","Type":"ContainerStarted","Data":"be5ea1c5f317588d73daae9a84f347098a3f02e0123de4e0880bfe6d7aaaaa65"}
Oct 30 14:17:12.217593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.217562    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" event={"ID":"f8631d1a-7032-493f-be3d-131ac728904a","Type":"ContainerStarted","Data":"d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2"}
Oct 30 14:17:12.217762 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.217616    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" event={"ID":"f8631d1a-7032-493f-be3d-131ac728904a","Type":"ContainerStarted","Data":"b6bca2e6b0cc241c6f6276500125b6c3d855dc0edf3e94b6f8166298b7c2655d"}
Oct 30 14:17:12.218924 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.218898    4530 generic.go:356] "Generic (PLEG): container finished" podID="e56997e4-1701-4fd2-8ddb-b12da63a113e" containerID="60cf8cb5c00ddf9ca17fdac8b288e67fc04652b7d254b32decb3a8dff63c7100" exitCode=137
Oct 30 14:17:12.219016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.218933    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9603-4db048"
Oct 30 14:17:12.219016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.218992    4530 scope.go:122] "RemoveContainer" containerID="60cf8cb5c00ddf9ca17fdac8b288e67fc04652b7d254b32decb3a8dff63c7100"
Oct 30 14:17:12.231442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.231419    4530 scope.go:122] "RemoveContainer" containerID="60cf8cb5c00ddf9ca17fdac8b288e67fc04652b7d254b32decb3a8dff63c7100"
Oct 30 14:17:12.231752 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:12.231728    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60cf8cb5c00ddf9ca17fdac8b288e67fc04652b7d254b32decb3a8dff63c7100\": container with ID starting with 60cf8cb5c00ddf9ca17fdac8b288e67fc04652b7d254b32decb3a8dff63c7100 not found: ID does not exist" containerID="60cf8cb5c00ddf9ca17fdac8b288e67fc04652b7d254b32decb3a8dff63c7100"
Oct 30 14:17:12.231843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.231758    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60cf8cb5c00ddf9ca17fdac8b288e67fc04652b7d254b32decb3a8dff63c7100"} err="failed to get container status \"60cf8cb5c00ddf9ca17fdac8b288e67fc04652b7d254b32decb3a8dff63c7100\": rpc error: code = NotFound desc = could not find container \"60cf8cb5c00ddf9ca17fdac8b288e67fc04652b7d254b32decb3a8dff63c7100\": container with ID starting with 60cf8cb5c00ddf9ca17fdac8b288e67fc04652b7d254b32decb3a8dff63c7100 not found: ID does not exist"
Oct 30 14:17:12.339615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.339576    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"]
Oct 30 14:17:12.341453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.341400    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" podStartSLOduration=2.341383501 podStartE2EDuration="2.341383501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:12.33596188 +0000 UTC m=+1697.367860820" watchObservedRunningTime="2026-10-30 14:17:12.341383501 +0000 UTC m=+1697.373282441"
Oct 30 14:17:12.347767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.346064    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:12.352816 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:12.351325    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-4db048\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e56997e4-1701-4fd2-8ddb-b12da63a113e" pod="opendatahub/test-curl-9603-4db048"
Oct 30 14:17:12.369045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.368212    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"]
Oct 30 14:17:12.369045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.368436    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-228454-serving-cert\""
Oct 30 14:17:12.433858 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:12.433755    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-4db048\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e56997e4-1701-4fd2-8ddb-b12da63a113e" pod="opendatahub/test-curl-9603-4db048"
Oct 30 14:17:12.570575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.569755    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:12.570575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.570261    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:12.570575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.570317    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9h6m\" (UniqueName: \"kubernetes.io/projected/fe54ee77-b40f-446a-9f20-55312643112c-kube-api-access-p9h6m\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:12.570575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.570374    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe54ee77-b40f-446a-9f20-55312643112c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:12.676559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.675744    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:12.676559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.676000    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:12.676559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.676039    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9h6m\" (UniqueName: \"kubernetes.io/projected/fe54ee77-b40f-446a-9f20-55312643112c-kube-api-access-p9h6m\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:12.676559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.676085    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe54ee77-b40f-446a-9f20-55312643112c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:12.692615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.676987    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe54ee77-b40f-446a-9f20-55312643112c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:12.692615 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:12.679658    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-serving-cert: secret "maas-api-e2e-shared-b-w3-228454-serving-cert" not found
Oct 30 14:17:12.692615 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:12.679731    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-maas-api-tls podName:fe54ee77-b40f-446a-9f20-55312643112c nodeName:}" failed. No retries permitted until 2026-10-30 14:17:13.179709618 +0000 UTC m=+1698.211608535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-maas-api-tls") pod "maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76" (UID: "fe54ee77-b40f-446a-9f20-55312643112c") : secret "maas-api-e2e-shared-b-w3-228454-serving-cert" not found
Oct 30 14:17:12.698212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.698177    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:12.716973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.716940    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:17:12.718213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.718182    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9h6m\" (UniqueName: \"kubernetes.io/projected/fe54ee77-b40f-446a-9f20-55312643112c-kube-api-access-p9h6m\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:12.724750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.724709    4530 patch_prober.go:29] interesting pod/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 14:17:12.724989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.724960    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 14:17:12.914943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.914903    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:12.921406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:12.921382    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:12.991481 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:12.991385    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9603-4db048\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="e56997e4-1701-4fd2-8ddb-b12da63a113e" pod="opendatahub/test-curl-9603-4db048"
Oct 30 14:17:13.117638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:13.117594    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bbbedd4-c15c-4dd5-8197-2087e9a6979a" path="/var/lib/kubelet/pods/8bbbedd4-c15c-4dd5-8197-2087e9a6979a/volumes"
Oct 30 14:17:13.118107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:13.118084    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56997e4-1701-4fd2-8ddb-b12da63a113e" path="/var/lib/kubelet/pods/e56997e4-1701-4fd2-8ddb-b12da63a113e/volumes"
Oct 30 14:17:13.208231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:13.208201    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:13.210800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:13.210765    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:13.228533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:13.228506    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:13.229489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:13.229469    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:17:13.276679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:13.274924    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:13.720366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:13.719911    4530 patch_prober.go:29] interesting pod/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 14:17:13.720366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:13.719986    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 14:17:13.794630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:13.790718    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"]
Oct 30 14:17:13.805952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:13.805837    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:17:14.253069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:14.253032    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76" event={"ID":"fe54ee77-b40f-446a-9f20-55312643112c","Type":"ContainerStarted","Data":"dc05b5de29502cd777e66e4ba5ad4447b784368c8ee4cfe71e07ec373a74f24c"}
Oct 30 14:17:14.622915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:14.622876    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:17:14.715973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:14.715930    4530 patch_prober.go:29] interesting pod/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 14:17:14.716247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:14.716217    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 14:17:15.264034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:15.263996    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76" event={"ID":"fe54ee77-b40f-446a-9f20-55312643112c","Type":"ContainerStarted","Data":"d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f"}
Oct 30 14:17:15.264640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:15.264343    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:15.330903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:15.330852    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76" podStartSLOduration=2.817378927 podStartE2EDuration="3.330837958s" totalImagesPullingTime="513.459031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:15.328764829 +0000 UTC m=+1700.360663770" watchObservedRunningTime="2026-10-30 14:17:15.330837958 +0000 UTC m=+1700.362736942"
Oct 30 14:17:15.713068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:15.712903    4530 patch_prober.go:29] interesting pod/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 14:17:15.713068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:15.712957    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 14:17:16.712104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:16.712062    4530 patch_prober.go:29] interesting pod/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 14:17:16.712519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:16.712125    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 14:17:17.712127 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:17.712091    4530 patch_prober.go:29] interesting pod/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 14:17:17.712528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:17.712151    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 14:17:18.716345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:18.716315    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:17:18.717509 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:18.717484    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:17:21.273773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:21.273735    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76" podUID="fe54ee77-b40f-446a-9f20-55312643112c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.173:8443/health\": dial tcp 10.133.0.173:8443: connect: connection refused"
Oct 30 14:17:22.043926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:22.043893    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"]
Oct 30 14:17:22.057951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:22.057921    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fbc72ea-5kswb"]
Oct 30 14:17:23.160698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:23.158330    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780aae55-c801-48ca-af16-80b5b8739b67" path="/var/lib/kubelet/pods/780aae55-c801-48ca-af16-80b5b8739b67/volumes"
Oct 30 14:17:24.485032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:24.485002    4530 generic.go:356] "Generic (PLEG): container finished" podID="fe54ee77-b40f-446a-9f20-55312643112c" containerID="d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f" exitCode=1
Oct 30 14:17:24.485348 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:24.485073    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76" event={"ID":"fe54ee77-b40f-446a-9f20-55312643112c","Type":"ContainerDied","Data":"d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f"}
Oct 30 14:17:24.485564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:24.485525    4530 scope.go:122] "RemoveContainer" containerID="d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f"
Oct 30 14:17:25.495195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:25.495163    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76" event={"ID":"fe54ee77-b40f-446a-9f20-55312643112c","Type":"ContainerStarted","Data":"2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b"}
Oct 30 14:17:25.495585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:25.495361    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:27.887588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.879875    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"]
Oct 30 14:17:27.887588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.884646    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:27.906592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.906565    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"]
Oct 30 14:17:27.913218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.913188    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d74808-openshift-default-dockercfg-z69lz\""
Oct 30 14:17:27.980558 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.980514    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-certs\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:27.980737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.980681    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqnc8\" (UniqueName: \"kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-kube-api-access-xqnc8\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:27.980737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.980724    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-token\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:27.980850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.980831    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ace7927-a93c-4a75-879d-be6e0b8aac79-istiod-ca-cert\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:27.980904 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.980883    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-envoy\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:27.980997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.980970    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-data\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:27.981043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.981019    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-socket\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:27.981089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.981047    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-credential-socket\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:27.981250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:27.981234    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-podinfo\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.099618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.083760    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqnc8\" (UniqueName: \"kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-kube-api-access-xqnc8\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.100916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.100881    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-token\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.101499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.101477    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ace7927-a93c-4a75-879d-be6e0b8aac79-istiod-ca-cert\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.102510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.102491    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-envoy\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.103104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.103084    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-data\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.103270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.103251    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-socket\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.103405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.103389    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-credential-socket\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.103708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.103688    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-podinfo\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.105243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.105215    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-certs\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.115791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.104345    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-credential-socket\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.115986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.102402    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ace7927-a93c-4a75-879d-be6e0b8aac79-istiod-ca-cert\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.116114 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.105037    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-data\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.116228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.112312    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-envoy\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.116336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.115458    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-certs\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.116474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.104100    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-socket\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.142190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.142098    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-podinfo\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.142973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.142947    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqnc8\" (UniqueName: \"kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-kube-api-access-xqnc8\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.155101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.155046    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-token\") pod \"e2e-api-a-d74808-openshift-default-5454f78567-g69q7\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") " pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.208176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.207887    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:28.916944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.916909    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"]
Oct 30 14:17:28.918382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.918343    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:28.918501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.918423    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:28.918501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:28.918469    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:29.559799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:29.559762    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" event={"ID":"8ace7927-a93c-4a75-879d-be6e0b8aac79","Type":"ContainerStarted","Data":"9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2"}
Oct 30 14:17:29.559999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:29.559825    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" event={"ID":"8ace7927-a93c-4a75-879d-be6e0b8aac79","Type":"ContainerStarted","Data":"4786d393688b15ba82dd9318a7ca31254e83ce85706596e9c6a95c36208d88ff"}
Oct 30 14:17:29.632521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:29.632470    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" podStartSLOduration=2.632456563 podStartE2EDuration="2.632456563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:29.627166222 +0000 UTC m=+1714.659065174" watchObservedRunningTime="2026-10-30 14:17:29.632456563 +0000 UTC m=+1714.664355574"
Oct 30 14:17:30.208618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:30.208585    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:30.214531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:30.214509    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:30.568366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:30.568336    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:30.569160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:30.569140    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:17:30.820010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:30.819925    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:17:31.065566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:31.062702    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"]
Oct 30 14:17:31.114299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:31.097507    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e8778e-v4rrf"]
Oct 30 14:17:31.145473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:31.145439    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8c779c-e58e-4b7c-b408-e7893410f4e7" path="/var/lib/kubelet/pods/2c8c779c-e58e-4b7c-b408-e7893410f4e7/volumes"
Oct 30 14:17:31.507491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:31.507409    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:17:38.049837 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:38.049804    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"]
Oct 30 14:17:38.064098 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:38.064069    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd44525d-g5zt4"]
Oct 30 14:17:39.117379 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:39.117337    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f388cf8f-159b-4884-9789-83e69fb9feb3" path="/var/lib/kubelet/pods/f388cf8f-159b-4884-9789-83e69fb9feb3/volumes"
Oct 30 14:17:41.353767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:41.353736    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"]
Oct 30 14:17:41.354240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:41.353989    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerName="istio-proxy" containerID="cri-o://a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3" gracePeriod=30
Oct 30 14:17:41.372773 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:41.372732    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:17:44.260588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.260554    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"]
Oct 30 14:17:44.297563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.293188    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:44.318613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.312977    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d727dc-serving-cert\""
Oct 30 14:17:44.318613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.315321    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-metrics-certs\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:44.318613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.315399    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:44.318613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.315437    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpgz8\" (UniqueName: \"kubernetes.io/projected/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-kube-api-access-gpgz8\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:44.318613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.315489    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-maas-api-tls\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:44.426577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.418013    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpgz8\" (UniqueName: \"kubernetes.io/projected/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-kube-api-access-gpgz8\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:44.426922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.426893    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-maas-api-tls\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:44.447341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.427399    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-metrics-certs\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:44.471640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.471609    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:44.472840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.470199    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"]
Oct 30 14:17:44.473229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.472521    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-metrics-certs\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:44.473359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.472620    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:44.473465 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:44.469005    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-serving-cert: secret "maas-api-e2e-api-a-d727dc-serving-cert" not found
Oct 30 14:17:44.473628 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:44.473616    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-maas-api-tls podName:3203b2df-5ed6-41af-b1a5-341fdbfd92a5 nodeName:}" failed. No retries permitted until 2026-10-30 14:17:44.973592901 +0000 UTC m=+1730.005491825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-maas-api-tls") pod "maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw" (UID: "3203b2df-5ed6-41af-b1a5-341fdbfd92a5") : secret "maas-api-e2e-api-a-d727dc-serving-cert" not found
Oct 30 14:17:44.645024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:44.644481    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpgz8\" (UniqueName: \"kubernetes.io/projected/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-kube-api-access-gpgz8\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:45.064332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:45.064295    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-maas-api-tls\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:45.066741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:45.066716    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-maas-api-tls\") pod \"maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:45.357649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:45.357556    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:45.666953 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:17:45.666919    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3203b2df_5ed6_41af_b1a5_341fdbfd92a5.slice/crio-3bb4917acb1ba8659444db58625aa4ee15ca15802cf0d506cf62877dd48786f4 WatchSource:0}: Error finding container 3bb4917acb1ba8659444db58625aa4ee15ca15802cf0d506cf62877dd48786f4: Status 404 returned error can't find the container with id 3bb4917acb1ba8659444db58625aa4ee15ca15802cf0d506cf62877dd48786f4
Oct 30 14:17:45.670309 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:45.670283    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"]
Oct 30 14:17:45.946843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:45.941655    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw" event={"ID":"3203b2df-5ed6-41af-b1a5-341fdbfd92a5","Type":"ContainerStarted","Data":"3bb4917acb1ba8659444db58625aa4ee15ca15802cf0d506cf62877dd48786f4"}
Oct 30 14:17:46.763854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.763823    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:17:46.824717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.824691    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-socket\") pod \"729aa207-6f8b-4283-8411-5bbd2eb20463\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") "
Oct 30 14:17:46.824884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.824799    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-envoy\") pod \"729aa207-6f8b-4283-8411-5bbd2eb20463\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") "
Oct 30 14:17:46.824884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.824880    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-certs\") pod \"729aa207-6f8b-4283-8411-5bbd2eb20463\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") "
Oct 30 14:17:46.825004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.824911    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-data\") pod \"729aa207-6f8b-4283-8411-5bbd2eb20463\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") "
Oct 30 14:17:46.825004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.824982    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-socket" pod "729aa207-6f8b-4283-8411-5bbd2eb20463" (UID: "729aa207-6f8b-4283-8411-5bbd2eb20463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:46.825120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.824994    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-token\" (UniqueName: \"kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-token\") pod \"729aa207-6f8b-4283-8411-5bbd2eb20463\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") "
Oct 30 14:17:46.825120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.825093    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-kube-api-access-t47k6\" (UniqueName: \"kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-kube-api-access-t47k6\") pod \"729aa207-6f8b-4283-8411-5bbd2eb20463\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") "
Oct 30 14:17:46.825227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.825207    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-credential-socket\") pod \"729aa207-6f8b-4283-8411-5bbd2eb20463\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") "
Oct 30 14:17:46.825292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.825230    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/729aa207-6f8b-4283-8411-5bbd2eb20463-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729aa207-6f8b-4283-8411-5bbd2eb20463-istiod-ca-cert\") pod \"729aa207-6f8b-4283-8411-5bbd2eb20463\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") "
Oct 30 14:17:46.825292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.825219    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-data" pod "729aa207-6f8b-4283-8411-5bbd2eb20463" (UID: "729aa207-6f8b-4283-8411-5bbd2eb20463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:46.825401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.825344    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-podinfo\") pod \"729aa207-6f8b-4283-8411-5bbd2eb20463\" (UID: \"729aa207-6f8b-4283-8411-5bbd2eb20463\") "
Oct 30 14:17:46.825459 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.825445    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-credential-socket" pod "729aa207-6f8b-4283-8411-5bbd2eb20463" (UID: "729aa207-6f8b-4283-8411-5bbd2eb20463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:46.825773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.825747    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-certs" pod "729aa207-6f8b-4283-8411-5bbd2eb20463" (UID: "729aa207-6f8b-4283-8411-5bbd2eb20463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:46.825773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.825754    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729aa207-6f8b-4283-8411-5bbd2eb20463-istiod-ca-cert" pod "729aa207-6f8b-4283-8411-5bbd2eb20463" (UID: "729aa207-6f8b-4283-8411-5bbd2eb20463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:17:46.826347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.826244    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:46.826347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.826269    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:46.826347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.826284    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:46.826347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.826297    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:46.826347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.826305    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729aa207-6f8b-4283-8411-5bbd2eb20463-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:46.827465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.827440    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-envoy" pod "729aa207-6f8b-4283-8411-5bbd2eb20463" (UID: "729aa207-6f8b-4283-8411-5bbd2eb20463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:17:46.827830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.827802    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-kube-api-access-t47k6" pod "729aa207-6f8b-4283-8411-5bbd2eb20463" (UID: "729aa207-6f8b-4283-8411-5bbd2eb20463"). InnerVolumeSpecName "kube-api-access-t47k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:17:46.827915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.827846    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-token" pod "729aa207-6f8b-4283-8411-5bbd2eb20463" (UID: "729aa207-6f8b-4283-8411-5bbd2eb20463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:17:46.827915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.827865    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-podinfo" pod "729aa207-6f8b-4283-8411-5bbd2eb20463" (UID: "729aa207-6f8b-4283-8411-5bbd2eb20463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:17:46.927791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.927761    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:46.927791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.927787    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:46.927791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.927795    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t47k6\" (UniqueName: \"kubernetes.io/projected/729aa207-6f8b-4283-8411-5bbd2eb20463-kube-api-access-t47k6\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:46.928026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.927805    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729aa207-6f8b-4283-8411-5bbd2eb20463-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:17:46.955390 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.955287    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw" event={"ID":"3203b2df-5ed6-41af-b1a5-341fdbfd92a5","Type":"ContainerStarted","Data":"9059ad836f80bec2d84afb3006ea0c1dc4b8de01614d69743594fd22b4e5850f"}
Oct 30 14:17:46.963743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.963711    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:46.975595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.975497    4530 generic.go:356] "Generic (PLEG): container finished" podID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerID="a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3" exitCode=0
Oct 30 14:17:46.975595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.975561    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"
Oct 30 14:17:46.975595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.975571    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" event={"ID":"729aa207-6f8b-4283-8411-5bbd2eb20463","Type":"ContainerDied","Data":"a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3"}
Oct 30 14:17:46.975854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.975626    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4" event={"ID":"729aa207-6f8b-4283-8411-5bbd2eb20463","Type":"ContainerDied","Data":"41933d5f202d16daee46b35564e77d998e69b267f87c9b6495c586533e3c5268"}
Oct 30 14:17:46.975854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.975650    4530 scope.go:122] "RemoveContainer" containerID="a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3"
Oct 30 14:17:46.992779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.992760    4530 scope.go:122] "RemoveContainer" containerID="a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3"
Oct 30 14:17:46.993059 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:46.993025    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3\": container with ID starting with a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3 not found: ID does not exist" containerID="a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3"
Oct 30 14:17:46.993133 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:46.993070    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3"} err="failed to get container status \"a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3\": rpc error: code = NotFound desc = could not find container \"a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3\": container with ID starting with a1f4465d4a04513cc85e1f32a70a0d0125ad65c12e4731967ed509754931a8a3 not found: ID does not exist"
Oct 30 14:17:47.030131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:47.030068    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw" podStartSLOduration=2.585951842 podStartE2EDuration="3.030056116s" totalImagesPullingTime="444.104274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:47.024465881 +0000 UTC m=+1732.056364821" watchObservedRunningTime="2026-10-30 14:17:47.030056116 +0000 UTC m=+1732.061955056"
Oct 30 14:17:47.089443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:47.089416    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"]
Oct 30 14:17:47.095134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:47.095110    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-lhgm4"]
Oct 30 14:17:47.116618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:47.116589    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" path="/var/lib/kubelet/pods/729aa207-6f8b-4283-8411-5bbd2eb20463/volumes"
Oct 30 14:17:49.039886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:49.039790    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"]
Oct 30 14:17:49.053152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:49.053125    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abfaa22a-sjpgm"]
Oct 30 14:17:49.116357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:49.116332    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f8a961-f83d-4ae8-848a-f0b97bee82b3" path="/var/lib/kubelet/pods/24f8a961-f83d-4ae8-848a-f0b97bee82b3/volumes"
Oct 30 14:17:51.558298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.558262    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"]
Oct 30 14:17:51.559471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.559446    4530 cpu_manager.go:414] "containerMap: removing container" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerName="istio-proxy"
Oct 30 14:17:51.559600 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.559475    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerName="istio-proxy"
Oct 30 14:17:51.559704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.559686    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="729aa207-6f8b-4283-8411-5bbd2eb20463" containerName="istio-proxy"
Oct 30 14:17:51.564293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.564270    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.586881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.586856    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"]
Oct 30 14:17:51.592957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.592889    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2b206b-openshift-default-dockercfg-x4llx\""
Oct 30 14:17:51.637460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.637424    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-podinfo\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.637654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.637491    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a4813f-02bc-4368-be3c-dcb62d5273cf-istiod-ca-cert\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.637724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.637640    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-data\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.637779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.637741    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-envoy\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.637818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.637775    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-socket\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.637862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.637829    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-token\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.637975 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.637955    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gdbs\" (UniqueName: \"kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-kube-api-access-2gdbs\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.638023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.638009    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-certs\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.638098 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.638083    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-credential-socket\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.739995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.739955    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-data\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.740168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.740027    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-envoy\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.740168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.740059    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-socket\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.740168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.740141    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-token\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.762892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.740207    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gdbs\" (UniqueName: \"kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-kube-api-access-2gdbs\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.762892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.740250    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-certs\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.762892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.740314    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-credential-socket\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.762892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.740614    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-podinfo\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.762892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.740669    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a4813f-02bc-4368-be3c-dcb62d5273cf-istiod-ca-cert\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.762892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.742393    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a4813f-02bc-4368-be3c-dcb62d5273cf-istiod-ca-cert\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.762892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.742761    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-data\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.762892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.748350    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-socket\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.762892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.748885    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-credential-socket\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.762892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.749154    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-certs\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.820199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.819221    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-podinfo\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.820990 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.820934    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-token\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.821397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.821357    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-envoy\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.841149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.841121    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gdbs\" (UniqueName: \"kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-kube-api-access-2gdbs\") pod \"e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") " pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:51.890864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:51.890828    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:52.329255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.329215    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"]
Oct 30 14:17:52.334802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.334785    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:52.355204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.355178    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"]
Oct 30 14:17:52.362840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.362813    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-972fee-serving-cert\""
Oct 30 14:17:52.416590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.416489    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw97r\" (UniqueName: \"kubernetes.io/projected/540d5e5d-ace9-431e-9a4f-982c27948672-kube-api-access-vw97r\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:52.416759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.416654    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-maas-api-tls\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:52.416832 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.416770    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540d5e5d-ace9-431e-9a4f-982c27948672-openshift-service-ca\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:52.416903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.416849    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-metrics-certs\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:52.428376 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:17:52.428351    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a4813f_02bc_4368_be3c_dcb62d5273cf.slice/crio-2ff31f615c878ddd4c2718fa4c74fc09d02bda5d048b8c02f04036abf6979d66 WatchSource:0}: Error finding container 2ff31f615c878ddd4c2718fa4c74fc09d02bda5d048b8c02f04036abf6979d66: Status 404 returned error can't find the container with id 2ff31f615c878ddd4c2718fa4c74fc09d02bda5d048b8c02f04036abf6979d66
Oct 30 14:17:52.431489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.431465    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"]
Oct 30 14:17:52.431710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.431682    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:52.431769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.431740    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:52.431769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.431765    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:17:52.536579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.529121    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw97r\" (UniqueName: \"kubernetes.io/projected/540d5e5d-ace9-431e-9a4f-982c27948672-kube-api-access-vw97r\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:52.536579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.529204    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-maas-api-tls\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:52.536579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.529295    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540d5e5d-ace9-431e-9a4f-982c27948672-openshift-service-ca\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:52.536579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.529367    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-metrics-certs\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:52.536579 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:52.535729    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-serving-cert: secret "maas-api-e2e-api-a-972fee-serving-cert" not found
Oct 30 14:17:52.536579 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:17:52.535840    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-maas-api-tls podName:540d5e5d-ace9-431e-9a4f-982c27948672 nodeName:}" failed. No retries permitted until 2026-10-30 14:17:53.035814464 +0000 UTC m=+1738.067713403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-maas-api-tls") pod "maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx" (UID: "540d5e5d-ace9-431e-9a4f-982c27948672") : secret "maas-api-e2e-api-a-972fee-serving-cert" not found
Oct 30 14:17:52.537382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.537107    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540d5e5d-ace9-431e-9a4f-982c27948672-openshift-service-ca\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:52.537382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.537150    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-metrics-certs\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:52.586611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.585301    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw97r\" (UniqueName: \"kubernetes.io/projected/540d5e5d-ace9-431e-9a4f-982c27948672-kube-api-access-vw97r\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:52.990801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:52.990715    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:17:53.037440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:53.037406    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-maas-api-tls\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:53.040190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:53.040164    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-maas-api-tls\") pod \"maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:53.126964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:53.126925    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" event={"ID":"75a4813f-02bc-4368-be3c-dcb62d5273cf","Type":"ContainerStarted","Data":"5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e"}
Oct 30 14:17:53.127145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:53.126980    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" event={"ID":"75a4813f-02bc-4368-be3c-dcb62d5273cf","Type":"ContainerStarted","Data":"2ff31f615c878ddd4c2718fa4c74fc09d02bda5d048b8c02f04036abf6979d66"}
Oct 30 14:17:53.184119 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:53.184066    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" podStartSLOduration=2.184051221 podStartE2EDuration="2.184051221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:53.178331404 +0000 UTC m=+1738.210230344" watchObservedRunningTime="2026-10-30 14:17:53.184051221 +0000 UTC m=+1738.215950160"
Oct 30 14:17:53.255485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:53.255417    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:53.621743 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:17:53.621705    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540d5e5d_ace9_431e_9a4f_982c27948672.slice/crio-284d0ec06782651e04008b61998d2b477c62a1353051773b78f17f7b577105c9 WatchSource:0}: Error finding container 284d0ec06782651e04008b61998d2b477c62a1353051773b78f17f7b577105c9: Status 404 returned error can't find the container with id 284d0ec06782651e04008b61998d2b477c62a1353051773b78f17f7b577105c9
Oct 30 14:17:53.624407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:53.624352    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"]
Oct 30 14:17:53.904634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:53.903779    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:53.929564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:53.927936    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:54.137260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:54.137233    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx" event={"ID":"540d5e5d-ace9-431e-9a4f-982c27948672","Type":"ContainerStarted","Data":"284d0ec06782651e04008b61998d2b477c62a1353051773b78f17f7b577105c9"}
Oct 30 14:17:54.137569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:54.137537    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:54.138420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:54.138401    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:17:55.148014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:55.147979    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx" event={"ID":"540d5e5d-ace9-431e-9a4f-982c27948672","Type":"ContainerStarted","Data":"93348344fed15e161469df22a267564b9d68db0b814e65ec640c1e06de840569"}
Oct 30 14:17:55.148409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:55.148228    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:17:55.196917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:55.196865    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx" podStartSLOduration=2.787588579 podStartE2EDuration="3.196852923s" totalImagesPullingTime="409.264344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:55.195515442 +0000 UTC m=+1740.227414394" watchObservedRunningTime="2026-10-30 14:17:55.196852923 +0000 UTC m=+1740.228751862"
Oct 30 14:17:56.409135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:56.409100    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"]
Oct 30 14:17:56.409619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:56.409356    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy" containerID="cri-o://78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d" gracePeriod=30
Oct 30 14:17:56.842987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:56.842952    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"]
Oct 30 14:17:56.843225 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:56.843200    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy" containerID="cri-o://aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6" gracePeriod=30
Oct 30 14:17:56.848342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:56.848314    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"]
Oct 30 14:17:56.848516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:56.848483    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy" containerID="cri-o://d0b0375ede5611c6eb6773998aba1949e408b41a2d46c471f8d022b2bbafd7f2" gracePeriod=30
Oct 30 14:17:56.860026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:56.859992    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"]
Oct 30 14:17:56.860298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:56.860250    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy" containerID="cri-o://861ae108b11109a183281056caf972fd1cfc63d394ec09e8156e3e63ba425344" gracePeriod=30
Oct 30 14:17:56.867999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:56.867970    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"]
Oct 30 14:17:56.868201 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:17:56.868177    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy" containerID="cri-o://733a5ee819a4e390e3e3798928e38561fafcdeb32d6aa1e4b24c8bdb56300ced" gracePeriod=30
Oct 30 14:18:00.845111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:00.845077    4530 scope.go:122] "RemoveContainer" containerID="751212252ddc65e1f9be25a08c6f6ae64e45fd827c78cd3be0fd6422588549b7"
Oct 30 14:18:00.856445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:00.856419    4530 scope.go:122] "RemoveContainer" containerID="745c0a9fb0c3a5f9854ace05aea34e9cf8caa9c219820e9189adc0764354285d"
Oct 30 14:18:00.866673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:00.866646    4530 scope.go:122] "RemoveContainer" containerID="f2eed58729d85b9dc97c463b0f43793b8d988306b80c7883340a525d5a21b526"
Oct 30 14:18:00.876424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:00.876402    4530 scope.go:122] "RemoveContainer" containerID="6b5f084ff1d1b3217b29182e201049e8ab710e8b2d3270f33b734e094a75e870"
Oct 30 14:18:00.885496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:00.885473    4530 scope.go:122] "RemoveContainer" containerID="cee9b0972ccb7cf02131cec2494d25c3a3572a5aa13f7b78be2a04a6e5df71e9"
Oct 30 14:18:00.895238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:00.895216    4530 scope.go:122] "RemoveContainer" containerID="7045891e5108d81bf5bc467078b61bd0e50c38d8cebd88b274a9b7aef8115345"
Oct 30 14:18:01.163238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.163176    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:18:01.687767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.687711    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:18:01.845741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.845710    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-socket\") pod \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") "
Oct 30 14:18:01.845741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.845745    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istiod-ca-cert\") pod \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") "
Oct 30 14:18:01.846342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.845770    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-envoy\") pod \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") "
Oct 30 14:18:01.846342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.845810    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-certs\") pod \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") "
Oct 30 14:18:01.846342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.845842    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-podinfo\") pod \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") "
Oct 30 14:18:01.846342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.845872    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-data\") pod \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") "
Oct 30 14:18:01.846342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.845986    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-kube-api-access-vfhhh\" (UniqueName: \"kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-kube-api-access-vfhhh\") pod \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") "
Oct 30 14:18:01.846342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.846032    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-token\" (UniqueName: \"kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-token\") pod \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") "
Oct 30 14:18:01.846342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.846077    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-credential-socket\") pod \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\" (UID: \"9b5c1a9e-e28f-4501-ab13-a4099e4220d3\") "
Oct 30 14:18:01.846342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.846113    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-socket" pod "9b5c1a9e-e28f-4501-ab13-a4099e4220d3" (UID: "9b5c1a9e-e28f-4501-ab13-a4099e4220d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:01.846342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.846137    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istiod-ca-cert" pod "9b5c1a9e-e28f-4501-ab13-a4099e4220d3" (UID: "9b5c1a9e-e28f-4501-ab13-a4099e4220d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:18:01.846342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.846266    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-certs" pod "9b5c1a9e-e28f-4501-ab13-a4099e4220d3" (UID: "9b5c1a9e-e28f-4501-ab13-a4099e4220d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:01.846950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.846481    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-credential-socket" pod "9b5c1a9e-e28f-4501-ab13-a4099e4220d3" (UID: "9b5c1a9e-e28f-4501-ab13-a4099e4220d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:01.846950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.846728    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-data" pod "9b5c1a9e-e28f-4501-ab13-a4099e4220d3" (UID: "9b5c1a9e-e28f-4501-ab13-a4099e4220d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:01.846950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.846940    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:01.847342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.846963    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:01.847342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.846977    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:01.847342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.846993    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:01.847342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.847006    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:01.849587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.848941    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-token" pod "9b5c1a9e-e28f-4501-ab13-a4099e4220d3" (UID: "9b5c1a9e-e28f-4501-ab13-a4099e4220d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:01.849587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.849054    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-podinfo" pod "9b5c1a9e-e28f-4501-ab13-a4099e4220d3" (UID: "9b5c1a9e-e28f-4501-ab13-a4099e4220d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:18:01.849587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.849232    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-kube-api-access-vfhhh" pod "9b5c1a9e-e28f-4501-ab13-a4099e4220d3" (UID: "9b5c1a9e-e28f-4501-ab13-a4099e4220d3"). InnerVolumeSpecName "kube-api-access-vfhhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:01.849587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.849368    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-envoy" pod "9b5c1a9e-e28f-4501-ab13-a4099e4220d3" (UID: "9b5c1a9e-e28f-4501-ab13-a4099e4220d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:01.948514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.948450    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:01.948514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.948476    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:01.948514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.948486    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfhhh\" (UniqueName: \"kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-kube-api-access-vfhhh\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:01.948514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:01.948496    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5c1a9e-e28f-4501-ab13-a4099e4220d3-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.161957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.161723    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:18:02.228521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.227914    4530 generic.go:356] "Generic (PLEG): container finished" podID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerID="78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d" exitCode=0
Oct 30 14:18:02.228521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.228050    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" event={"ID":"9b5c1a9e-e28f-4501-ab13-a4099e4220d3","Type":"ContainerDied","Data":"78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d"}
Oct 30 14:18:02.228521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.228105    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6" event={"ID":"9b5c1a9e-e28f-4501-ab13-a4099e4220d3","Type":"ContainerDied","Data":"c12b797e96a6411616061c4819367af74b81c8a301e407d348c48e39a62d5e2e"}
Oct 30 14:18:02.228521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.228138    4530 scope.go:122] "RemoveContainer" containerID="78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d"
Oct 30 14:18:02.228521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.228323    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"
Oct 30 14:18:02.244834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.244798    4530 generic.go:356] "Generic (PLEG): container finished" podID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerID="861ae108b11109a183281056caf972fd1cfc63d394ec09e8156e3e63ba425344" exitCode=0
Oct 30 14:18:02.245011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.244958    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" event={"ID":"7663d949-64c3-46ed-a16d-abf54a2b78ec","Type":"ContainerDied","Data":"861ae108b11109a183281056caf972fd1cfc63d394ec09e8156e3e63ba425344"}
Oct 30 14:18:02.247487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.246990    4530 generic.go:356] "Generic (PLEG): container finished" podID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerID="aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6" exitCode=0
Oct 30 14:18:02.247487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.247205    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" event={"ID":"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b","Type":"ContainerDied","Data":"aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6"}
Oct 30 14:18:02.247487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.247249    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv" event={"ID":"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b","Type":"ContainerDied","Data":"1923d6d00444593dd9d11fb5c436540def6464a094470c1902471c823881f219"}
Oct 30 14:18:02.247487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.247335    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"
Oct 30 14:18:02.250560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.250477    4530 generic.go:356] "Generic (PLEG): container finished" podID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerID="733a5ee819a4e390e3e3798928e38561fafcdeb32d6aa1e4b24c8bdb56300ced" exitCode=0
Oct 30 14:18:02.251088 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.250953    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" event={"ID":"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5","Type":"ContainerDied","Data":"733a5ee819a4e390e3e3798928e38561fafcdeb32d6aa1e4b24c8bdb56300ced"}
Oct 30 14:18:02.254078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.253510    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-credential-socket\") pod \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") "
Oct 30 14:18:02.254078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.253629    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-kube-api-access-4qcxb\" (UniqueName: \"kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-kube-api-access-4qcxb\") pod \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") "
Oct 30 14:18:02.254078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.253708    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-envoy\") pod \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") "
Oct 30 14:18:02.254078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.253813    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-podinfo\") pod \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") "
Oct 30 14:18:02.254078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.253862    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-token\" (UniqueName: \"kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-token\") pod \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") "
Oct 30 14:18:02.254078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.253862    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-credential-socket" pod "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" (UID: "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.254078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.253889    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-data\") pod \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") "
Oct 30 14:18:02.254078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.253963    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-socket\") pod \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") "
Oct 30 14:18:02.254078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.253993    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istiod-ca-cert\") pod \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") "
Oct 30 14:18:02.254078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.254083    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-certs\") pod \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\" (UID: \"d28a89c2-3e68-4cb3-a3ff-14a147b7e72b\") "
Oct 30 14:18:02.255510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.254822    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.255510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.255168    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-certs" pod "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" (UID: "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.255510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.255242    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-socket" pod "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" (UID: "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.255722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.255583    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istiod-ca-cert" pod "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" (UID: "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:18:02.256371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.256308    4530 generic.go:356] "Generic (PLEG): container finished" podID="9da0e269-c913-4d18-8831-b9e89e184737" containerID="d0b0375ede5611c6eb6773998aba1949e408b41a2d46c471f8d022b2bbafd7f2" exitCode=0
Oct 30 14:18:02.256371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.256329    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-data" pod "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" (UID: "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.256371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.256340    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" event={"ID":"9da0e269-c913-4d18-8831-b9e89e184737","Type":"ContainerDied","Data":"d0b0375ede5611c6eb6773998aba1949e408b41a2d46c471f8d022b2bbafd7f2"}
Oct 30 14:18:02.266982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.266961    4530 scope.go:122] "RemoveContainer" containerID="963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b"
Oct 30 14:18:02.270231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.269223    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-envoy" pod "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" (UID: "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.270231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.269402    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-kube-api-access-4qcxb" pod "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" (UID: "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b"). InnerVolumeSpecName "kube-api-access-4qcxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:02.270231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.269475    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-token" pod "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" (UID: "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:02.271510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.271488    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-podinfo" pod "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" (UID: "d28a89c2-3e68-4cb3-a3ff-14a147b7e72b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:18:02.286044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.282735    4530 scope.go:122] "RemoveContainer" containerID="78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d"
Oct 30 14:18:02.286044 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:18:02.284259    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d\": container with ID starting with 78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d not found: ID does not exist" containerID="78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d"
Oct 30 14:18:02.286044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.284293    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d"} err="failed to get container status \"78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d\": rpc error: code = NotFound desc = could not find container \"78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d\": container with ID starting with 78e4ca93363d3e0b13d3b4238af26b3318ab46e87c6e57db7418ccd3910b5d2d not found: ID does not exist"
Oct 30 14:18:02.286044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.284311    4530 scope.go:122] "RemoveContainer" containerID="963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b"
Oct 30 14:18:02.286044 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:18:02.285058    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b\": container with ID starting with 963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b not found: ID does not exist" containerID="963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b"
Oct 30 14:18:02.286044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.285089    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b"} err="failed to get container status \"963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b\": rpc error: code = NotFound desc = could not find container \"963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b\": container with ID starting with 963fe4f117b2df9321ceef2fba97d362f421d77fc471b34d6ecd591a3e25692b not found: ID does not exist"
Oct 30 14:18:02.286044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.285105    4530 scope.go:122] "RemoveContainer" containerID="aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6"
Oct 30 14:18:02.294590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.292972    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:18:02.298204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.298187    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:18:02.299370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.299349    4530 scope.go:122] "RemoveContainer" containerID="aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6"
Oct 30 14:18:02.299659 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:18:02.299634    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6\": container with ID starting with aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6 not found: ID does not exist" containerID="aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6"
Oct 30 14:18:02.299750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.299666    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6"} err="failed to get container status \"aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6\": rpc error: code = NotFound desc = could not find container \"aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6\": container with ID starting with aa0247f45099052857273aa7efa72d4c2ae5aacb4d7c36e17e7ceb228ae0a1a6 not found: ID does not exist"
Oct 30 14:18:02.299750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.299684    4530 scope.go:122] "RemoveContainer" containerID="420afa151307904bbc62416a306c9a02e3604e2529c4cfeba67812c50ed01035"
Oct 30 14:18:02.301112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.301094    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:18:02.346372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.346336    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"]
Oct 30 14:18:02.351122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.351097    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-w8th6"]
Oct 30 14:18:02.355702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.355677    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-data\") pod \"9da0e269-c913-4d18-8831-b9e89e184737\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") "
Oct 30 14:18:02.355831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.355734    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-socket\") pod \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") "
Oct 30 14:18:02.355831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.355788    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da0e269-c913-4d18-8831-b9e89e184737-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da0e269-c913-4d18-8831-b9e89e184737-istiod-ca-cert\") pod \"9da0e269-c913-4d18-8831-b9e89e184737\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") "
Oct 30 14:18:02.355831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.355827    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istiod-ca-cert\") pod \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") "
Oct 30 14:18:02.356003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.355860    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-data\") pod \"7663d949-64c3-46ed-a16d-abf54a2b78ec\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") "
Oct 30 14:18:02.356003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.355916    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-certs\") pod \"7663d949-64c3-46ed-a16d-abf54a2b78ec\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") "
Oct 30 14:18:02.356003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.355976    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-istio-token\" (UniqueName: \"kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-istio-token\") pod \"9da0e269-c913-4d18-8831-b9e89e184737\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") "
Oct 30 14:18:02.356003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.355980    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-data" pod "9da0e269-c913-4d18-8831-b9e89e184737" (UID: "9da0e269-c913-4d18-8831-b9e89e184737"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.356222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356009    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-podinfo\") pod \"7663d949-64c3-46ed-a16d-abf54a2b78ec\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") "
Oct 30 14:18:02.356222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356080    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-kube-api-access-2gq4j\" (UniqueName: \"kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-kube-api-access-2gq4j\") pod \"9da0e269-c913-4d18-8831-b9e89e184737\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") "
Oct 30 14:18:02.356222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356099    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-socket" pod "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" (UID: "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.356222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356113    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7663d949-64c3-46ed-a16d-abf54a2b78ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7663d949-64c3-46ed-a16d-abf54a2b78ec-istiod-ca-cert\") pod \"7663d949-64c3-46ed-a16d-abf54a2b78ec\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") "
Oct 30 14:18:02.356222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356140    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-kube-api-access-fjgt6\" (UniqueName: \"kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-kube-api-access-fjgt6\") pod \"7663d949-64c3-46ed-a16d-abf54a2b78ec\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") "
Oct 30 14:18:02.356222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356221    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-credential-socket\") pod \"9da0e269-c913-4d18-8831-b9e89e184737\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") "
Oct 30 14:18:02.356567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356253    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-certs" pod "7663d949-64c3-46ed-a16d-abf54a2b78ec" (UID: "7663d949-64c3-46ed-a16d-abf54a2b78ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.356567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356269    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-credential-socket\") pod \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") "
Oct 30 14:18:02.356567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356294    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da0e269-c913-4d18-8831-b9e89e184737-istiod-ca-cert" pod "9da0e269-c913-4d18-8831-b9e89e184737" (UID: "9da0e269-c913-4d18-8831-b9e89e184737"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:18:02.356567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356322    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istiod-ca-cert" pod "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" (UID: "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:18:02.356567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356334    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-certs\") pod \"9da0e269-c913-4d18-8831-b9e89e184737\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") "
Oct 30 14:18:02.356567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356380    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-envoy\") pod \"9da0e269-c913-4d18-8831-b9e89e184737\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") "
Oct 30 14:18:02.356567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356413    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-certs\") pod \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") "
Oct 30 14:18:02.356567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356444    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-envoy\") pod \"7663d949-64c3-46ed-a16d-abf54a2b78ec\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") "
Oct 30 14:18:02.356567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356482    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-data" pod "7663d949-64c3-46ed-a16d-abf54a2b78ec" (UID: "7663d949-64c3-46ed-a16d-abf54a2b78ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.357061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356711    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-credential-socket" pod "9da0e269-c913-4d18-8831-b9e89e184737" (UID: "9da0e269-c913-4d18-8831-b9e89e184737"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.357126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.357084    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7663d949-64c3-46ed-a16d-abf54a2b78ec-istiod-ca-cert" pod "7663d949-64c3-46ed-a16d-abf54a2b78ec" (UID: "7663d949-64c3-46ed-a16d-abf54a2b78ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:18:02.357222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.356486    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-token\" (UniqueName: \"kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-token\") pod \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") "
Oct 30 14:18:02.357304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.357259    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9da0e269-c913-4d18-8831-b9e89e184737-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da0e269-c913-4d18-8831-b9e89e184737-istio-podinfo\") pod \"9da0e269-c913-4d18-8831-b9e89e184737\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") "
Oct 30 14:18:02.357391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.357343    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-data\") pod \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") "
Oct 30 14:18:02.357391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.357361    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-credential-socket" pod "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" (UID: "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.357495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.357388    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-podinfo\") pod \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") "
Oct 30 14:18:02.357612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.357588    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-certs" pod "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" (UID: "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.358579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.358518    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-envoy\") pod \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") "
Oct 30 14:18:02.358675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.358594    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-token\" (UniqueName: \"kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-token\") pod \"7663d949-64c3-46ed-a16d-abf54a2b78ec\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") "
Oct 30 14:18:02.358675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.358633    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-socket\") pod \"9da0e269-c913-4d18-8831-b9e89e184737\" (UID: \"9da0e269-c913-4d18-8831-b9e89e184737\") "
Oct 30 14:18:02.358802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.358677    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-credential-socket\") pod \"7663d949-64c3-46ed-a16d-abf54a2b78ec\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") "
Oct 30 14:18:02.358802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.358742    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-kube-api-access-25qxj\" (UniqueName: \"kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-kube-api-access-25qxj\") pod \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\" (UID: \"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5\") "
Oct 30 14:18:02.358882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.358817    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-socket\") pod \"7663d949-64c3-46ed-a16d-abf54a2b78ec\" (UID: \"7663d949-64c3-46ed-a16d-abf54a2b78ec\") "
Oct 30 14:18:02.359808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359788    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.359808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359807    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359817    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359826    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7663d949-64c3-46ed-a16d-abf54a2b78ec-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359835    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359842    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359851    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359860    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359868    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359878    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359891    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qcxb\" (UniqueName: \"kubernetes.io/projected/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-kube-api-access-4qcxb\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359903    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359915    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359927    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359941    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da0e269-c913-4d18-8831-b9e89e184737-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359953    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359966    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359968    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-podinfo" pod "7663d949-64c3-46ed-a16d-abf54a2b78ec" (UID: "7663d949-64c3-46ed-a16d-abf54a2b78ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:18:02.360023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.359978    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.361004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.360167    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-socket" pod "7663d949-64c3-46ed-a16d-abf54a2b78ec" (UID: "7663d949-64c3-46ed-a16d-abf54a2b78ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.361004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.360311    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-istio-token" pod "9da0e269-c913-4d18-8831-b9e89e184737" (UID: "9da0e269-c913-4d18-8831-b9e89e184737"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:02.361004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.360513    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-certs" pod "9da0e269-c913-4d18-8831-b9e89e184737" (UID: "9da0e269-c913-4d18-8831-b9e89e184737"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.361004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.360628    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-kube-api-access-fjgt6" pod "7663d949-64c3-46ed-a16d-abf54a2b78ec" (UID: "7663d949-64c3-46ed-a16d-abf54a2b78ec"). InnerVolumeSpecName "kube-api-access-fjgt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:02.361004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.360664    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-token" pod "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" (UID: "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:02.361004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.360846    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-envoy" pod "7663d949-64c3-46ed-a16d-abf54a2b78ec" (UID: "7663d949-64c3-46ed-a16d-abf54a2b78ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.361004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.360900    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-data" pod "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" (UID: "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.361355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.361092    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-credential-socket" pod "7663d949-64c3-46ed-a16d-abf54a2b78ec" (UID: "7663d949-64c3-46ed-a16d-abf54a2b78ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.361355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.361244    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-socket" pod "9da0e269-c913-4d18-8831-b9e89e184737" (UID: "9da0e269-c913-4d18-8831-b9e89e184737"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.361467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.361416    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-envoy" pod "9da0e269-c913-4d18-8831-b9e89e184737" (UID: "9da0e269-c913-4d18-8831-b9e89e184737"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.361897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.361863    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-kube-api-access-2gq4j" pod "9da0e269-c913-4d18-8831-b9e89e184737" (UID: "9da0e269-c913-4d18-8831-b9e89e184737"). InnerVolumeSpecName "kube-api-access-2gq4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:02.362755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.362729    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-envoy" pod "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" (UID: "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:02.363296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.363275    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-podinfo" pod "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" (UID: "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:18:02.363427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.363407    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-token" pod "7663d949-64c3-46ed-a16d-abf54a2b78ec" (UID: "7663d949-64c3-46ed-a16d-abf54a2b78ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:02.363473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.363406    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9da0e269-c913-4d18-8831-b9e89e184737-istio-podinfo" pod "9da0e269-c913-4d18-8831-b9e89e184737" (UID: "9da0e269-c913-4d18-8831-b9e89e184737"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:18:02.363512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.363480    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-kube-api-access-25qxj" pod "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" (UID: "7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5"). InnerVolumeSpecName "kube-api-access-25qxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:02.462280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462245    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.462416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462286    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.462416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462303    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25qxj\" (UniqueName: \"kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-kube-api-access-25qxj\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.462416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462316    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.462416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462331    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.462416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462345    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.462416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462357    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gq4j\" (UniqueName: \"kubernetes.io/projected/9da0e269-c913-4d18-8831-b9e89e184737-kube-api-access-2gq4j\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.462416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462371    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjgt6\" (UniqueName: \"kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-kube-api-access-fjgt6\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.462416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462384    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.462416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462397    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da0e269-c913-4d18-8831-b9e89e184737-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.462416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462408    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.462416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462419    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.463038 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462431    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da0e269-c913-4d18-8831-b9e89e184737-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.463038 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462443    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.463038 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462455    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.463038 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462467    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.463038 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.462481    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7663d949-64c3-46ed-a16d-abf54a2b78ec-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:02.609006 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.608713    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"]
Oct 30 14:18:02.700691 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:02.697206    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-2pvwv"]
Oct 30 14:18:03.066438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.066407    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"]
Oct 30 14:18:03.070976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.070950    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ded77db-tlqwl"]
Oct 30 14:18:03.119171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.119148    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" path="/var/lib/kubelet/pods/9b5c1a9e-e28f-4501-ab13-a4099e4220d3/volumes"
Oct 30 14:18:03.119673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.119658    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" path="/var/lib/kubelet/pods/d28a89c2-3e68-4cb3-a3ff-14a147b7e72b/volumes"
Oct 30 14:18:03.120112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.120099    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf3233f-658d-458c-b6e3-a1ac8b215c95" path="/var/lib/kubelet/pods/ebf3233f-658d-458c-b6e3-a1ac8b215c95/volumes"
Oct 30 14:18:03.266446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.266416    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"
Oct 30 14:18:03.266446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.266430    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658" event={"ID":"7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5","Type":"ContainerDied","Data":"c8b6388da989d70205a76a03efb7ebb4f09da926ee3021ea9ae3fff5e83231ff"}
Oct 30 14:18:03.266666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.266485    4530 scope.go:122] "RemoveContainer" containerID="733a5ee819a4e390e3e3798928e38561fafcdeb32d6aa1e4b24c8bdb56300ced"
Oct 30 14:18:03.268060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.268044    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"
Oct 30 14:18:03.268171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.268037    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj" event={"ID":"9da0e269-c913-4d18-8831-b9e89e184737","Type":"ContainerDied","Data":"28812084eb156cb486311cfa1b246ab658a84cbb06aefb95be291d1c36ef800b"}
Oct 30 14:18:03.271327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.271306    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs" event={"ID":"7663d949-64c3-46ed-a16d-abf54a2b78ec","Type":"ContainerDied","Data":"22600474b58647dbd2ca1f0ca1fe8a123d75d30f7b345a5755cd1fc5d0172eb6"}
Oct 30 14:18:03.271422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.271330    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"
Oct 30 14:18:03.278622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.278601    4530 scope.go:122] "RemoveContainer" containerID="d0b0375ede5611c6eb6773998aba1949e408b41a2d46c471f8d022b2bbafd7f2"
Oct 30 14:18:03.288283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.288261    4530 scope.go:122] "RemoveContainer" containerID="861ae108b11109a183281056caf972fd1cfc63d394ec09e8156e3e63ba425344"
Oct 30 14:18:03.364389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.364336    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"]
Oct 30 14:18:03.368551 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.368523    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-tg5tj"]
Oct 30 14:18:03.408933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.408912    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"]
Oct 30 14:18:03.417045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.417013    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9c658"]
Oct 30 14:18:03.457898 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.457876    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"]
Oct 30 14:18:03.463328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:03.463307    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-9pkxs"]
Oct 30 14:18:05.116785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:05.116750    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" path="/var/lib/kubelet/pods/7663d949-64c3-46ed-a16d-abf54a2b78ec/volumes"
Oct 30 14:18:05.117448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:05.117427    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" path="/var/lib/kubelet/pods/7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5/volumes"
Oct 30 14:18:05.118004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:05.117981    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da0e269-c913-4d18-8831-b9e89e184737" path="/var/lib/kubelet/pods/9da0e269-c913-4d18-8831-b9e89e184737/volumes"
Oct 30 14:18:08.028624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:08.018178    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:18:11.369696 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:18:11.369650    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:18:11.371945 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:11.371920    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"]
Oct 30 14:18:11.372322 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:11.372300    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq" podUID="59b0d295-010d-4193-8d2d-c9dc02150bd8" containerName="istio-proxy" containerID="cri-o://a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09" gracePeriod=30
Oct 30 14:18:11.438569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:11.414875    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"]
Oct 30 14:18:11.438569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:11.415126    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" containerName="istio-proxy" containerID="cri-o://be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89" gracePeriod=30
Oct 30 14:18:12.140011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:12.139979    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"]
Oct 30 14:18:12.140423 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:12.140393    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy" containerID="cri-o://f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9" gracePeriod=30
Oct 30 14:18:12.152933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:12.152817    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"]
Oct 30 14:18:12.153059 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:12.153019    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy" containerID="cri-o://7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03" gracePeriod=30
Oct 30 14:18:12.159854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:12.159830    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"]
Oct 30 14:18:12.160148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:12.160127    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" podUID="0e166404-62ee-4c62-83b7-ba6375714a5b" containerName="istio-proxy" containerID="cri-o://f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338" gracePeriod=30
Oct 30 14:18:12.164679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:12.164656    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"]
Oct 30 14:18:12.164832 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:12.164817    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy" containerID="cri-o://8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50" gracePeriod=30
Oct 30 14:18:13.038887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:13.038844    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"]
Oct 30 14:18:13.043255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:13.043228    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9760220e-hlqr4"]
Oct 30 14:18:13.136716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:13.136684    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88ca0ca-4b55-4052-be9e-ed18a12a9d3a" path="/var/lib/kubelet/pods/c88ca0ca-4b55-4052-be9e-ed18a12a9d3a/volumes"
Oct 30 14:18:13.747588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:13.746351    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"]
Oct 30 14:18:13.747588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:13.746623    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf" podUID="1c41d290-df32-406e-badd-688c5e316f53" containerName="maas-api" containerID="cri-o://def33dafede5ec5b8fc50d42b94d81072ac0280ec80cbc749eb176322dc0f332" gracePeriod=30
Oct 30 14:18:14.956831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:14.956340    4530 generic.go:356] "Generic (PLEG): container finished" podID="1c41d290-df32-406e-badd-688c5e316f53" containerID="def33dafede5ec5b8fc50d42b94d81072ac0280ec80cbc749eb176322dc0f332" exitCode=0
Oct 30 14:18:14.957594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:14.957368    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf" event={"ID":"1c41d290-df32-406e-badd-688c5e316f53","Type":"ContainerDied","Data":"def33dafede5ec5b8fc50d42b94d81072ac0280ec80cbc749eb176322dc0f332"}
Oct 30 14:18:15.048449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.048420    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:18:15.141880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.141841    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-maas-api-tls\") pod \"1c41d290-df32-406e-badd-688c5e316f53\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") "
Oct 30 14:18:15.142073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.141916    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c41d290-df32-406e-badd-688c5e316f53-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c41d290-df32-406e-badd-688c5e316f53-openshift-service-ca\") pod \"1c41d290-df32-406e-badd-688c5e316f53\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") "
Oct 30 14:18:15.142241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.142224    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c41d290-df32-406e-badd-688c5e316f53-kube-api-access-7rdhc\" (UniqueName: \"kubernetes.io/projected/1c41d290-df32-406e-badd-688c5e316f53-kube-api-access-7rdhc\") pod \"1c41d290-df32-406e-badd-688c5e316f53\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") "
Oct 30 14:18:15.142324 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.142290    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-metrics-certs\") pod \"1c41d290-df32-406e-badd-688c5e316f53\" (UID: \"1c41d290-df32-406e-badd-688c5e316f53\") "
Oct 30 14:18:15.142447 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.142400    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c41d290-df32-406e-badd-688c5e316f53-openshift-service-ca" pod "1c41d290-df32-406e-badd-688c5e316f53" (UID: "1c41d290-df32-406e-badd-688c5e316f53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:18:15.143885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.143851    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c41d290-df32-406e-badd-688c5e316f53-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:15.145592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.145524    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-metrics-certs" pod "1c41d290-df32-406e-badd-688c5e316f53" (UID: "1c41d290-df32-406e-badd-688c5e316f53"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:18:15.146003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.145977    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-maas-api-tls" pod "1c41d290-df32-406e-badd-688c5e316f53" (UID: "1c41d290-df32-406e-badd-688c5e316f53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:18:15.147019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.146989    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c41d290-df32-406e-badd-688c5e316f53-kube-api-access-7rdhc" pod "1c41d290-df32-406e-badd-688c5e316f53" (UID: "1c41d290-df32-406e-badd-688c5e316f53"). InnerVolumeSpecName "kube-api-access-7rdhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:15.249217 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.249119    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rdhc\" (UniqueName: \"kubernetes.io/projected/1c41d290-df32-406e-badd-688c5e316f53-kube-api-access-7rdhc\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:15.249398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.249258    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:15.249398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.249281    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c41d290-df32-406e-badd-688c5e316f53-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:15.968187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.968155    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"
Oct 30 14:18:15.968602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.968222    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf" event={"ID":"1c41d290-df32-406e-badd-688c5e316f53","Type":"ContainerDied","Data":"18a8575de2fbabbdc0aa59110f1740542db00f0b623e389d17d547a601180edd"}
Oct 30 14:18:15.968602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:15.968269    4530 scope.go:122] "RemoveContainer" containerID="def33dafede5ec5b8fc50d42b94d81072ac0280ec80cbc749eb176322dc0f332"
Oct 30 14:18:16.056575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.056517    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"]
Oct 30 14:18:16.070421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.070393    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-qfzbf"]
Oct 30 14:18:16.404657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.404623    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"]
Oct 30 14:18:16.405694 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.405662    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy"
Oct 30 14:18:16.405694 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.405689    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy"
Oct 30 14:18:16.405912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.405707    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy"
Oct 30 14:18:16.405912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.405715    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy"
Oct 30 14:18:16.405912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.405731    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy"
Oct 30 14:18:16.405912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.405736    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy"
Oct 30 14:18:16.405912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.405749    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy"
Oct 30 14:18:16.405912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.405753    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy"
Oct 30 14:18:16.405912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.405769    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy"
Oct 30 14:18:16.405912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.405775    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy"
Oct 30 14:18:16.405912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.405819    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1c41d290-df32-406e-badd-688c5e316f53" containerName="maas-api"
Oct 30 14:18:16.405912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.405828    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c41d290-df32-406e-badd-688c5e316f53" containerName="maas-api"
Oct 30 14:18:16.406478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.406005    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28a89c2-3e68-4cb3-a3ff-14a147b7e72b" containerName="istio-proxy"
Oct 30 14:18:16.406478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.406025    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy"
Oct 30 14:18:16.406478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.406036    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7663d949-64c3-46ed-a16d-abf54a2b78ec" containerName="istio-proxy"
Oct 30 14:18:16.406478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.406050    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c41d290-df32-406e-badd-688c5e316f53" containerName="maas-api"
Oct 30 14:18:16.406478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.406063    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy"
Oct 30 14:18:16.406478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.406073    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy"
Oct 30 14:18:16.406478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.406084    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af3ba0a-bb15-4ae7-8542-bbf4eb7674f5" containerName="istio-proxy"
Oct 30 14:18:16.410030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.410006    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.431779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.431748    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d74808-serving-cert\""
Oct 30 14:18:16.433871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.433762    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"]
Oct 30 14:18:16.466633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.466157    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-metrics-certs\") pod \"maas-api-e2e-api-a-d74808-764fbb749c-tcqgb\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.466633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.466251    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3091155-b159-4764-a9e2-8668aeeb286f-openshift-service-ca\") pod \"maas-api-e2e-api-a-d74808-764fbb749c-tcqgb\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.466633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.466377    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5kcv\" (UniqueName: \"kubernetes.io/projected/b3091155-b159-4764-a9e2-8668aeeb286f-kube-api-access-k5kcv\") pod \"maas-api-e2e-api-a-d74808-764fbb749c-tcqgb\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.466633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.466566    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-maas-api-tls\") pod \"maas-api-e2e-api-a-d74808-764fbb749c-tcqgb\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.568387 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.568349    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kcv\" (UniqueName: \"kubernetes.io/projected/b3091155-b159-4764-a9e2-8668aeeb286f-kube-api-access-k5kcv\") pod \"maas-api-e2e-api-a-d74808-764fbb749c-tcqgb\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.568582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.568474    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-maas-api-tls\") pod \"maas-api-e2e-api-a-d74808-764fbb749c-tcqgb\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.568582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.568567    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-metrics-certs\") pod \"maas-api-e2e-api-a-d74808-764fbb749c-tcqgb\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.568708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.568613    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3091155-b159-4764-a9e2-8668aeeb286f-openshift-service-ca\") pod \"maas-api-e2e-api-a-d74808-764fbb749c-tcqgb\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.569337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.569314    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3091155-b159-4764-a9e2-8668aeeb286f-openshift-service-ca\") pod \"maas-api-e2e-api-a-d74808-764fbb749c-tcqgb\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.580561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.576226    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-metrics-certs\") pod \"maas-api-e2e-api-a-d74808-764fbb749c-tcqgb\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.580561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.576715    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-maas-api-tls\") pod \"maas-api-e2e-api-a-d74808-764fbb749c-tcqgb\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.600855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.598797    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5kcv\" (UniqueName: \"kubernetes.io/projected/b3091155-b159-4764-a9e2-8668aeeb286f-kube-api-access-k5kcv\") pod \"maas-api-e2e-api-a-d74808-764fbb749c-tcqgb\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.801147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.801111    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:18:16.855385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.855354    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:16.890709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.890242    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b0d295-010d-4193-8d2d-c9dc02150bd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b0d295-010d-4193-8d2d-c9dc02150bd8-istiod-ca-cert\") pod \"59b0d295-010d-4193-8d2d-c9dc02150bd8\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") "
Oct 30 14:18:16.890709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.890313    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-kube-api-access-bzlq5\" (UniqueName: \"kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-kube-api-access-bzlq5\") pod \"59b0d295-010d-4193-8d2d-c9dc02150bd8\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") "
Oct 30 14:18:16.890709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.890360    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-credential-socket\") pod \"59b0d295-010d-4193-8d2d-c9dc02150bd8\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") "
Oct 30 14:18:16.890709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.890443    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-token\" (UniqueName: \"kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-token\") pod \"59b0d295-010d-4193-8d2d-c9dc02150bd8\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") "
Oct 30 14:18:16.890709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.890479    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-certs\") pod \"59b0d295-010d-4193-8d2d-c9dc02150bd8\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") "
Oct 30 14:18:16.890709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.890531    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-envoy\") pod \"59b0d295-010d-4193-8d2d-c9dc02150bd8\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") "
Oct 30 14:18:16.890709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.890607    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b0d295-010d-4193-8d2d-c9dc02150bd8-istiod-ca-cert" pod "59b0d295-010d-4193-8d2d-c9dc02150bd8" (UID: "59b0d295-010d-4193-8d2d-c9dc02150bd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:18:16.890709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.890626    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-podinfo\") pod \"59b0d295-010d-4193-8d2d-c9dc02150bd8\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") "
Oct 30 14:18:16.890709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.890685    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-data\") pod \"59b0d295-010d-4193-8d2d-c9dc02150bd8\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") "
Oct 30 14:18:16.891314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.890906    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-socket\") pod \"59b0d295-010d-4193-8d2d-c9dc02150bd8\" (UID: \"59b0d295-010d-4193-8d2d-c9dc02150bd8\") "
Oct 30 14:18:16.891882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.891658    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b0d295-010d-4193-8d2d-c9dc02150bd8-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:16.891996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.891911    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-data" pod "59b0d295-010d-4193-8d2d-c9dc02150bd8" (UID: "59b0d295-010d-4193-8d2d-c9dc02150bd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:16.892132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.892094    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-socket" pod "59b0d295-010d-4193-8d2d-c9dc02150bd8" (UID: "59b0d295-010d-4193-8d2d-c9dc02150bd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:16.892132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.892107    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-credential-socket" pod "59b0d295-010d-4193-8d2d-c9dc02150bd8" (UID: "59b0d295-010d-4193-8d2d-c9dc02150bd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:16.892302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.892204    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-certs" pod "59b0d295-010d-4193-8d2d-c9dc02150bd8" (UID: "59b0d295-010d-4193-8d2d-c9dc02150bd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:16.896663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.896618    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-podinfo" pod "59b0d295-010d-4193-8d2d-c9dc02150bd8" (UID: "59b0d295-010d-4193-8d2d-c9dc02150bd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:18:16.897989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.897963    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-kube-api-access-bzlq5" pod "59b0d295-010d-4193-8d2d-c9dc02150bd8" (UID: "59b0d295-010d-4193-8d2d-c9dc02150bd8"). InnerVolumeSpecName "kube-api-access-bzlq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:16.898076 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.898006    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-token" pod "59b0d295-010d-4193-8d2d-c9dc02150bd8" (UID: "59b0d295-010d-4193-8d2d-c9dc02150bd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:16.898792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.898374    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:18:16.899870 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.899830    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-envoy" pod "59b0d295-010d-4193-8d2d-c9dc02150bd8" (UID: "59b0d295-010d-4193-8d2d-c9dc02150bd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:16.993656 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.993618    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:16.994142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.994121    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzlq5\" (UniqueName: \"kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-kube-api-access-bzlq5\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:16.994284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.994269    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:16.994403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.994389    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:16.994514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.994502    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:16.994625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.994614    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:16.994707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.994697    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:16.994803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.994791    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b0d295-010d-4193-8d2d-c9dc02150bd8-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:16.996936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.996905    4530 generic.go:356] "Generic (PLEG): container finished" podID="46b8b223-4999-4a21-a275-093515f6ccfe" containerID="be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89" exitCode=0
Oct 30 14:18:16.997039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.997005    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" event={"ID":"46b8b223-4999-4a21-a275-093515f6ccfe","Type":"ContainerDied","Data":"be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89"}
Oct 30 14:18:16.997112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.997098    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9" event={"ID":"46b8b223-4999-4a21-a275-093515f6ccfe","Type":"ContainerDied","Data":"c5c952300600f8cc0241c691999fa2d777915408746aaccbd24d7126422f6d1f"}
Oct 30 14:18:16.997166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.997140    4530 scope.go:122] "RemoveContainer" containerID="be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89"
Oct 30 14:18:16.997250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:16.997237    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"
Oct 30 14:18:17.000292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.000263    4530 generic.go:356] "Generic (PLEG): container finished" podID="59b0d295-010d-4193-8d2d-c9dc02150bd8" containerID="a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09" exitCode=0
Oct 30 14:18:17.000504 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.000482    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq" event={"ID":"59b0d295-010d-4193-8d2d-c9dc02150bd8","Type":"ContainerDied","Data":"a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09"}
Oct 30 14:18:17.000608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.000533    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq" event={"ID":"59b0d295-010d-4193-8d2d-c9dc02150bd8","Type":"ContainerDied","Data":"f1615640ba755b6a0a5b2102ded9cb3ce3a54d53c69f3a3dcd2df30386b4f2f4"}
Oct 30 14:18:17.000788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.000688    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"
Oct 30 14:18:17.027473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.027442    4530 scope.go:122] "RemoveContainer" containerID="be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89"
Oct 30 14:18:17.027865 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:18:17.027844    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89\": container with ID starting with be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89 not found: ID does not exist" containerID="be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89"
Oct 30 14:18:17.027932 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.027875    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89"} err="failed to get container status \"be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89\": rpc error: code = NotFound desc = could not find container \"be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89\": container with ID starting with be9c6b643ed971fe3b642793efdee071139ffe904c0fe56646bdceb4f2001a89 not found: ID does not exist"
Oct 30 14:18:17.027932 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.027891    4530 scope.go:122] "RemoveContainer" containerID="a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09"
Oct 30 14:18:17.041854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.041831    4530 scope.go:122] "RemoveContainer" containerID="a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09"
Oct 30 14:18:17.042278 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:18:17.042253    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09\": container with ID starting with a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09 not found: ID does not exist" containerID="a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09"
Oct 30 14:18:17.042435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.042411    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09"} err="failed to get container status \"a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09\": rpc error: code = NotFound desc = could not find container \"a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09\": container with ID starting with a33310fbee952643df83f819a1283f567fc5f7bd2e2652c4aed327731adecf09 not found: ID does not exist"
Oct 30 14:18:17.091202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.091172    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"]
Oct 30 14:18:17.095676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.095640    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-data\") pod \"46b8b223-4999-4a21-a275-093515f6ccfe\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") "
Oct 30 14:18:17.095831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.095706    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-envoy\") pod \"46b8b223-4999-4a21-a275-093515f6ccfe\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") "
Oct 30 14:18:17.095831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.095783    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-socket\") pod \"46b8b223-4999-4a21-a275-093515f6ccfe\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") "
Oct 30 14:18:17.095986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.095969    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-credential-socket\") pod \"46b8b223-4999-4a21-a275-093515f6ccfe\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") "
Oct 30 14:18:17.096046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.096025    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-kube-api-access-cnwz2\" (UniqueName: \"kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-kube-api-access-cnwz2\") pod \"46b8b223-4999-4a21-a275-093515f6ccfe\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") "
Oct 30 14:18:17.096104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.096052    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-istio-token\" (UniqueName: \"kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-istio-token\") pod \"46b8b223-4999-4a21-a275-093515f6ccfe\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") "
Oct 30 14:18:17.096104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.096087    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b8b223-4999-4a21-a275-093515f6ccfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b8b223-4999-4a21-a275-093515f6ccfe-istiod-ca-cert\") pod \"46b8b223-4999-4a21-a275-093515f6ccfe\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") "
Oct 30 14:18:17.096207 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.096127    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-socket" pod "46b8b223-4999-4a21-a275-093515f6ccfe" (UID: "46b8b223-4999-4a21-a275-093515f6ccfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.096276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.096256    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46b8b223-4999-4a21-a275-093515f6ccfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b8b223-4999-4a21-a275-093515f6ccfe-istio-podinfo\") pod \"46b8b223-4999-4a21-a275-093515f6ccfe\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") "
Oct 30 14:18:17.096355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.096339    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-certs\") pod \"46b8b223-4999-4a21-a275-093515f6ccfe\" (UID: \"46b8b223-4999-4a21-a275-093515f6ccfe\") "
Oct 30 14:18:17.096473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.096455    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b8b223-4999-4a21-a275-093515f6ccfe-istiod-ca-cert" pod "46b8b223-4999-4a21-a275-093515f6ccfe" (UID: "46b8b223-4999-4a21-a275-093515f6ccfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:18:17.096728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.096709    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-credential-socket" pod "46b8b223-4999-4a21-a275-093515f6ccfe" (UID: "46b8b223-4999-4a21-a275-093515f6ccfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.096941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.096918    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-certs" pod "46b8b223-4999-4a21-a275-093515f6ccfe" (UID: "46b8b223-4999-4a21-a275-093515f6ccfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.097156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.097139    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.097220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.097163    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.097220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.097178    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b8b223-4999-4a21-a275-093515f6ccfe-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.097220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.097192    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.097386 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.097298    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-data" pod "46b8b223-4999-4a21-a275-093515f6ccfe" (UID: "46b8b223-4999-4a21-a275-093515f6ccfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.099615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.099591    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-4vjwq"]
Oct 30 14:18:17.099843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.099817    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-kube-api-access-cnwz2" pod "46b8b223-4999-4a21-a275-093515f6ccfe" (UID: "46b8b223-4999-4a21-a275-093515f6ccfe"). InnerVolumeSpecName "kube-api-access-cnwz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:17.100807 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.100780    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-envoy" pod "46b8b223-4999-4a21-a275-093515f6ccfe" (UID: "46b8b223-4999-4a21-a275-093515f6ccfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.100937 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.100913    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-istio-token" pod "46b8b223-4999-4a21-a275-093515f6ccfe" (UID: "46b8b223-4999-4a21-a275-093515f6ccfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:17.101982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.101959    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46b8b223-4999-4a21-a275-093515f6ccfe-istio-podinfo" pod "46b8b223-4999-4a21-a275-093515f6ccfe" (UID: "46b8b223-4999-4a21-a275-093515f6ccfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:18:17.119070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.119041    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c41d290-df32-406e-badd-688c5e316f53" path="/var/lib/kubelet/pods/1c41d290-df32-406e-badd-688c5e316f53/volumes"
Oct 30 14:18:17.119840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.119821    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b0d295-010d-4193-8d2d-c9dc02150bd8" path="/var/lib/kubelet/pods/59b0d295-010d-4193-8d2d-c9dc02150bd8/volumes"
Oct 30 14:18:17.191249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.191214    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"]
Oct 30 14:18:17.198435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.198407    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnwz2\" (UniqueName: \"kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-kube-api-access-cnwz2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.198435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.198437    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b8b223-4999-4a21-a275-093515f6ccfe-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.198596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.198446    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b8b223-4999-4a21-a275-093515f6ccfe-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.198596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.198455    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.198596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.198466    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b8b223-4999-4a21-a275-093515f6ccfe-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.377641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.377560    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"]
Oct 30 14:18:17.468465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.468436    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-t5cm9"]
Oct 30 14:18:17.493340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.493026    4530 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 14:18:17.493340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.493086    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" podUID="0e166404-62ee-4c62-83b7-ba6375714a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 14:18:17.596048 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.595766    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:18:17.654712 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.645102    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.671992    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istiod-ca-cert\") pod \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672059    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-socket\") pod \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672110    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-podinfo\") pod \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672195    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-token\" (UniqueName: \"kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-token\") pod \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672240    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-kube-api-access-2zmd8\" (UniqueName: \"kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-kube-api-access-2zmd8\") pod \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672365    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-socket" pod "cfb0ff55-8edc-43e6-b24e-6325a97efd1c" (UID: "cfb0ff55-8edc-43e6-b24e-6325a97efd1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672388    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-token\" (UniqueName: \"kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-token\") pod \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672442    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-podinfo\") pod \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672481    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-certs\") pod \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672514    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-kube-api-access-2c687\" (UniqueName: \"kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-kube-api-access-2c687\") pod \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672627    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-data\") pod \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672689    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-socket\") pod \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672774    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-envoy\") pod \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672801    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-credential-socket\") pod \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672847    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-envoy\") pod \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672922    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-credential-socket\") pod \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\" (UID: \"cfb0ff55-8edc-43e6-b24e-6325a97efd1c\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672970    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istiod-ca-cert\") pod \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.673061    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-data\") pod \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.673185    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-certs\") pod \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\" (UID: \"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5\") "
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.672383    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istiod-ca-cert" pod "cfb0ff55-8edc-43e6-b24e-6325a97efd1c" (UID: "cfb0ff55-8edc-43e6-b24e-6325a97efd1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.673657    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-socket" pod "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" (UID: "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.673943    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.673969    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.673989    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.674208    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-certs" pod "cfb0ff55-8edc-43e6-b24e-6325a97efd1c" (UID: "cfb0ff55-8edc-43e6-b24e-6325a97efd1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.674678    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-data" pod "cfb0ff55-8edc-43e6-b24e-6325a97efd1c" (UID: "cfb0ff55-8edc-43e6-b24e-6325a97efd1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.675297    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istiod-ca-cert" pod "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" (UID: "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.675556    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-credential-socket" pod "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" (UID: "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.675922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.675744    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-certs" pod "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" (UID: "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.677620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.676103    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-data" pod "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" (UID: "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.677620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.676221    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-credential-socket" pod "cfb0ff55-8edc-43e6-b24e-6325a97efd1c" (UID: "cfb0ff55-8edc-43e6-b24e-6325a97efd1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.678681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.678650    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-envoy" pod "cfb0ff55-8edc-43e6-b24e-6325a97efd1c" (UID: "cfb0ff55-8edc-43e6-b24e-6325a97efd1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.680882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.680849    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-envoy" pod "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" (UID: "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.680991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.680931    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-token" pod "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" (UID: "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:17.682456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.682433    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-podinfo" pod "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" (UID: "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:18:17.682625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.682608    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-kube-api-access-2c687" pod "cfb0ff55-8edc-43e6-b24e-6325a97efd1c" (UID: "cfb0ff55-8edc-43e6-b24e-6325a97efd1c"). InnerVolumeSpecName "kube-api-access-2c687". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:17.682810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.682786    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-token" pod "cfb0ff55-8edc-43e6-b24e-6325a97efd1c" (UID: "cfb0ff55-8edc-43e6-b24e-6325a97efd1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:17.683044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.683020    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-podinfo" pod "cfb0ff55-8edc-43e6-b24e-6325a97efd1c" (UID: "cfb0ff55-8edc-43e6-b24e-6325a97efd1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:18:17.684464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.684427    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-kube-api-access-2zmd8" pod "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" (UID: "c18ce5d6-43d9-4210-a54d-329f8ae1e3d5"). InnerVolumeSpecName "kube-api-access-2zmd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:17.714446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.714114    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:18:17.721768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.721736    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774436    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-socket\") pod \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774494    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-certs\") pod \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774585    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-certs\") pod \"0e166404-62ee-4c62-83b7-ba6375714a5b\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774617    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-kube-api-access-r4b68\" (UniqueName: \"kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-kube-api-access-r4b68\") pod \"0e166404-62ee-4c62-83b7-ba6375714a5b\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774649    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-envoy\") pod \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774700    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-token\" (UniqueName: \"kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-token\") pod \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774740    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-credential-socket\") pod \"0e166404-62ee-4c62-83b7-ba6375714a5b\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774760    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-socket" pod "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" (UID: "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774793    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e166404-62ee-4c62-83b7-ba6375714a5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e166404-62ee-4c62-83b7-ba6375714a5b-istiod-ca-cert\") pod \"0e166404-62ee-4c62-83b7-ba6375714a5b\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774824    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-token\" (UniqueName: \"kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-token\") pod \"0e166404-62ee-4c62-83b7-ba6375714a5b\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774863    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-podinfo\") pod \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774904    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-socket\") pod \"0e166404-62ee-4c62-83b7-ba6375714a5b\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.774946    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istiod-ca-cert\") pod \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.775101    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-envoy\") pod \"0e166404-62ee-4c62-83b7-ba6375714a5b\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.775161    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-credential-socket\") pod \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.775239    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-data\") pod \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.775293    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-data\") pod \"0e166404-62ee-4c62-83b7-ba6375714a5b\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.775348    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-podinfo\") pod \"0e166404-62ee-4c62-83b7-ba6375714a5b\" (UID: \"0e166404-62ee-4c62-83b7-ba6375714a5b\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.775384    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-kube-api-access-qxxk2\" (UniqueName: \"kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-kube-api-access-qxxk2\") pod \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\" (UID: \"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a\") "
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776198    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776222    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776237    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776254    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776267    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776281    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776296    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776308    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776321    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zmd8\" (UniqueName: \"kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-kube-api-access-2zmd8\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776334    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776346    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776360    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776374    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c687\" (UniqueName: \"kubernetes.io/projected/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-kube-api-access-2c687\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776387    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb0ff55-8edc-43e6-b24e-6325a97efd1c-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776564    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.778072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.776578    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.780004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.779857    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-credential-socket" pod "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" (UID: "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.780004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.779871    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-data" pod "0e166404-62ee-4c62-83b7-ba6375714a5b" (UID: "0e166404-62ee-4c62-83b7-ba6375714a5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.780004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.779956    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-socket" pod "0e166404-62ee-4c62-83b7-ba6375714a5b" (UID: "0e166404-62ee-4c62-83b7-ba6375714a5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.780167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.780076    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-data" pod "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" (UID: "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.780167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.780102    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istiod-ca-cert" pod "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" (UID: "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:18:17.780314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.780275    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-certs" pod "0e166404-62ee-4c62-83b7-ba6375714a5b" (UID: "0e166404-62ee-4c62-83b7-ba6375714a5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.780441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.780417    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e166404-62ee-4c62-83b7-ba6375714a5b-istiod-ca-cert" pod "0e166404-62ee-4c62-83b7-ba6375714a5b" (UID: "0e166404-62ee-4c62-83b7-ba6375714a5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:18:17.780738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.780711    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-credential-socket" pod "0e166404-62ee-4c62-83b7-ba6375714a5b" (UID: "0e166404-62ee-4c62-83b7-ba6375714a5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.781083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.781061    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-certs" pod "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" (UID: "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.782241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.781682    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-kube-api-access-r4b68" pod "0e166404-62ee-4c62-83b7-ba6375714a5b" (UID: "0e166404-62ee-4c62-83b7-ba6375714a5b"). InnerVolumeSpecName "kube-api-access-r4b68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:17.787600 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.787115    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-kube-api-access-qxxk2" pod "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" (UID: "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a"). InnerVolumeSpecName "kube-api-access-qxxk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:17.788522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.788255    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-token" pod "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" (UID: "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:17.789854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.789832    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-token" pod "0e166404-62ee-4c62-83b7-ba6375714a5b" (UID: "0e166404-62ee-4c62-83b7-ba6375714a5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:18:17.790570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.790528    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-podinfo" pod "0e166404-62ee-4c62-83b7-ba6375714a5b" (UID: "0e166404-62ee-4c62-83b7-ba6375714a5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:18:17.793956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.793920    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-podinfo" pod "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" (UID: "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:18:17.794180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.794147    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-envoy" pod "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" (UID: "cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.805265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.805238    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-envoy" pod "0e166404-62ee-4c62-83b7-ba6375714a5b" (UID: "0e166404-62ee-4c62-83b7-ba6375714a5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878073    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878104    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878117    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4b68\" (UniqueName: \"kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-kube-api-access-r4b68\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878130    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878142    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878154    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878166    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e166404-62ee-4c62-83b7-ba6375714a5b-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878177    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878189    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878200    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878214    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878225    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878237    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878248    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878259    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878271    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e166404-62ee-4c62-83b7-ba6375714a5b-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:17.878312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:17.878284    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxxk2\" (UniqueName: \"kubernetes.io/projected/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a-kube-api-access-qxxk2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:18:18.124680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.124638    4530 generic.go:356] "Generic (PLEG): container finished" podID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerID="7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03" exitCode=0
Oct 30 14:18:18.125356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.125325    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" event={"ID":"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5","Type":"ContainerDied","Data":"7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03"}
Oct 30 14:18:18.125835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.125817    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf" event={"ID":"c18ce5d6-43d9-4210-a54d-329f8ae1e3d5","Type":"ContainerDied","Data":"80e83622c800a63e95ff53ddee72b00c9c7f9321c63b4ad9d722c8fd7702a0ac"}
Oct 30 14:18:18.125979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.125964    4530 scope.go:122] "RemoveContainer" containerID="7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03"
Oct 30 14:18:18.126135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.125757    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"
Oct 30 14:18:18.137808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.136300    4530 generic.go:356] "Generic (PLEG): container finished" podID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerID="8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50" exitCode=0
Oct 30 14:18:18.137808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.136495    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" event={"ID":"cfb0ff55-8edc-43e6-b24e-6325a97efd1c","Type":"ContainerDied","Data":"8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50"}
Oct 30 14:18:18.137808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.136582    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h" event={"ID":"cfb0ff55-8edc-43e6-b24e-6325a97efd1c","Type":"ContainerDied","Data":"fd3722400e74a621471e03ac7b9c4227e27684e7cf3dd68af8b8badb90061c15"}
Oct 30 14:18:18.137808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.136691    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"
Oct 30 14:18:18.170881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.169750    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb" event={"ID":"b3091155-b159-4764-a9e2-8668aeeb286f","Type":"ContainerStarted","Data":"b75cd4b86089e5c79419ac68c04d034cc063587f63e79a5ffa20061426d9bc1d"}
Oct 30 14:18:18.170881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.169824    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb" event={"ID":"b3091155-b159-4764-a9e2-8668aeeb286f","Type":"ContainerStarted","Data":"75bb538ba90ecf89c9d224152b336bc1b449ac31a19b548cf10bed9902de1572"}
Oct 30 14:18:18.170881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.170805    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:18.175770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.175735    4530 generic.go:356] "Generic (PLEG): container finished" podID="0e166404-62ee-4c62-83b7-ba6375714a5b" containerID="f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338" exitCode=0
Oct 30 14:18:18.175935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.175889    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" event={"ID":"0e166404-62ee-4c62-83b7-ba6375714a5b","Type":"ContainerDied","Data":"f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338"}
Oct 30 14:18:18.176007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.175943    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr" event={"ID":"0e166404-62ee-4c62-83b7-ba6375714a5b","Type":"ContainerDied","Data":"9d6b38638f98a6e4d4a3040229617bba0dc3ae984b922f599d3362748b49e8b2"}
Oct 30 14:18:18.176075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.176034    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"
Oct 30 14:18:18.178684 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.178663    4530 scope.go:122] "RemoveContainer" containerID="7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03"
Oct 30 14:18:18.179185 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:18:18.179050    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03\": container with ID starting with 7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03 not found: ID does not exist" containerID="7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03"
Oct 30 14:18:18.179185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.179085    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03"} err="failed to get container status \"7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03\": rpc error: code = NotFound desc = could not find container \"7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03\": container with ID starting with 7370c557399827ffcd2be580296168a9d9d16961731c2d18871b5c514992bf03 not found: ID does not exist"
Oct 30 14:18:18.179185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.179119    4530 scope.go:122] "RemoveContainer" containerID="8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50"
Oct 30 14:18:18.180843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.180818    4530 generic.go:356] "Generic (PLEG): container finished" podID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerID="f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9" exitCode=0
Oct 30 14:18:18.180952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.180910    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" event={"ID":"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a","Type":"ContainerDied","Data":"f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9"}
Oct 30 14:18:18.181024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.180949    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb" event={"ID":"cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a","Type":"ContainerDied","Data":"814c08df4e961853edc4092428edaff788cc5d5315afe5f5a22c1edd87e5512b"}
Oct 30 14:18:18.181024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.181020    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"
Oct 30 14:18:18.196288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.196266    4530 scope.go:122] "RemoveContainer" containerID="8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50"
Oct 30 14:18:18.196711 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:18:18.196692    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50\": container with ID starting with 8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50 not found: ID does not exist" containerID="8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50"
Oct 30 14:18:18.196790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.196740    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50"} err="failed to get container status \"8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50\": rpc error: code = NotFound desc = could not find container \"8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50\": container with ID starting with 8532dd5af0a2569eae154018f822552097e3752516bd1358b956c9a5dae57f50 not found: ID does not exist"
Oct 30 14:18:18.196790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.196759    4530 scope.go:122] "RemoveContainer" containerID="f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338"
Oct 30 14:18:18.210988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.210535    4530 scope.go:122] "RemoveContainer" containerID="f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338"
Oct 30 14:18:18.210988 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:18:18.210936    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338\": container with ID starting with f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338 not found: ID does not exist" containerID="f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338"
Oct 30 14:18:18.210988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.210973    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338"} err="failed to get container status \"f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338\": rpc error: code = NotFound desc = could not find container \"f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338\": container with ID starting with f8ff74a46f3515b3d187ffe72c81f11f3c7a3c71946b6773264c2f0e4c84f338 not found: ID does not exist"
Oct 30 14:18:18.211254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.210996    4530 scope.go:122] "RemoveContainer" containerID="f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9"
Oct 30 14:18:18.223805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.223786    4530 scope.go:122] "RemoveContainer" containerID="0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873"
Oct 30 14:18:18.247053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.247021    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"]
Oct 30 14:18:18.247334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.247315    4530 scope.go:122] "RemoveContainer" containerID="f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9"
Oct 30 14:18:18.247797 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:18:18.247766    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9\": container with ID starting with f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9 not found: ID does not exist" containerID="f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9"
Oct 30 14:18:18.247910 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.247805    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9"} err="failed to get container status \"f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9\": rpc error: code = NotFound desc = could not find container \"f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9\": container with ID starting with f730d8dcd822c0b684ccdc26f60c93dd6b9cb3fe00de911cf030a993eb6de4a9 not found: ID does not exist"
Oct 30 14:18:18.247910 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.247831    4530 scope.go:122] "RemoveContainer" containerID="0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873"
Oct 30 14:18:18.248164 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:18:18.248147    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873\": container with ID starting with 0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873 not found: ID does not exist" containerID="0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873"
Oct 30 14:18:18.248231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.248170    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873"} err="failed to get container status \"0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873\": rpc error: code = NotFound desc = could not find container \"0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873\": container with ID starting with 0fe65fe8168d214c507081bbf378ab9259e4ec2901ef5c6d651ec2de08bf8873 not found: ID does not exist"
Oct 30 14:18:18.254683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.254662    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vnlbf"]
Oct 30 14:18:18.294132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.294094    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"]
Oct 30 14:18:18.298992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.298958    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wr62h"]
Oct 30 14:18:18.345530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.345473    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb" podStartSLOduration=1.8760523340000002 podStartE2EDuration="2.345458502s" totalImagesPullingTime="469.406168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:18.341137949 +0000 UTC m=+1763.373036889" watchObservedRunningTime="2026-10-30 14:18:18.345458502 +0000 UTC m=+1763.377357474"
Oct 30 14:18:18.389534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.389452    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"]
Oct 30 14:18:18.396475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.396441    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-vkszr"]
Oct 30 14:18:18.441519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.437707    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"]
Oct 30 14:18:18.442696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:18.442673    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-ns4nb"]
Oct 30 14:18:19.132230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:19.132187    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e166404-62ee-4c62-83b7-ba6375714a5b" path="/var/lib/kubelet/pods/0e166404-62ee-4c62-83b7-ba6375714a5b/volumes"
Oct 30 14:18:19.133112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:19.133085    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" path="/var/lib/kubelet/pods/46b8b223-4999-4a21-a275-093515f6ccfe/volumes"
Oct 30 14:18:19.134217 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:19.134192    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" path="/var/lib/kubelet/pods/c18ce5d6-43d9-4210-a54d-329f8ae1e3d5/volumes"
Oct 30 14:18:19.134705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:19.134686    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" path="/var/lib/kubelet/pods/cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a/volumes"
Oct 30 14:18:19.135159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:19.135145    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" path="/var/lib/kubelet/pods/cfb0ff55-8edc-43e6-b24e-6325a97efd1c/volumes"
Oct 30 14:18:21.225878 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:21.225847    4530 generic.go:356] "Generic (PLEG): container finished" podID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerID="b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b" exitCode=0
Oct 30 14:18:21.227379 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:21.225924    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" event={"ID":"af3e9eea-3660-4b16-8cc8-afe897b3b3b2","Type":"ContainerDied","Data":"b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b"}
Oct 30 14:18:21.227379 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:21.226347    4530 scope.go:122] "RemoveContainer" containerID="b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b"
Oct 30 14:18:21.228102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:21.228075    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:18:21.228158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:21.228146    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:18:21.228194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:21.228179    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:18:22.270006 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:22.269965    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" event={"ID":"af3e9eea-3660-4b16-8cc8-afe897b3b3b2","Type":"ContainerStarted","Data":"1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd"}
Oct 30 14:18:22.275012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:22.274985    4530 generic.go:356] "Generic (PLEG): container finished" podID="f8631d1a-7032-493f-be3d-131ac728904a" containerID="d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2" exitCode=0
Oct 30 14:18:22.275171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:22.275095    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" event={"ID":"f8631d1a-7032-493f-be3d-131ac728904a","Type":"ContainerDied","Data":"d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2"}
Oct 30 14:18:22.275471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:22.275431    4530 scope.go:122] "RemoveContainer" containerID="d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2"
Oct 30 14:18:22.277934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:22.277885    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:18:22.278156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:22.278133    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:18:22.278302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:22.278285    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:18:22.657485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:22.655051    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:18:22.657485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:22.656694    4530 patch_prober.go:29] interesting pod/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 14:18:22.657485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:22.656747    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 14:18:23.327567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:23.325229    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" event={"ID":"f8631d1a-7032-493f-be3d-131ac728904a","Type":"ContainerStarted","Data":"a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c"}
Oct 30 14:18:23.551445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:23.551403    4530 patch_prober.go:29] interesting pod/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 14:18:23.552965 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:23.552292    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 14:18:23.923648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:23.923374    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:18:23.923648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:23.923600    4530 patch_prober.go:29] interesting pod/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 14:18:23.923899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:23.923648    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 14:18:24.095564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:24.091625    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:18:24.555112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:24.555077    4530 patch_prober.go:29] interesting pod/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 14:18:24.556023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:24.555982    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 14:18:24.918618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:24.917924    4530 patch_prober.go:29] interesting pod/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 14:18:24.918618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:24.917988    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 14:18:25.228123 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:25.228040    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:18:25.549459 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:25.549427    4530 patch_prober.go:29] interesting pod/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 14:18:25.549656 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:25.549490    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 14:18:25.916619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:25.915665    4530 patch_prober.go:29] interesting pod/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 14:18:25.916619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:25.915726    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 14:18:26.549420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:26.548946    4530 patch_prober.go:29] interesting pod/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 14:18:26.549420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:26.549007    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 14:18:26.929462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:26.928942    4530 patch_prober.go:29] interesting pod/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 14:18:26.929462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:26.929007    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 14:18:27.549494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.549454    4530 patch_prober.go:29] interesting pod/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 14:18:27.549691 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.549515    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.639310    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"]
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640296    4530 cpu_manager.go:414] "containerMap: removing container" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640316    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640338    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640346    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640372    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640381    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640412    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640420    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da0e269-c913-4d18-8831-b9e89e184737" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640429    4530 cpu_manager.go:414] "containerMap: removing container" podUID="59b0d295-010d-4193-8d2d-c9dc02150bd8" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640436    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b0d295-010d-4193-8d2d-c9dc02150bd8" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640464    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640471    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640486    4530 cpu_manager.go:414] "containerMap: removing container" podUID="0e166404-62ee-4c62-83b7-ba6375714a5b" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640492    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e166404-62ee-4c62-83b7-ba6375714a5b" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640504    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640511    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640681    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb0ff55-8edc-43e6-b24e-6325a97efd1c" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640695    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e166404-62ee-4c62-83b7-ba6375714a5b" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640704    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5c1a9e-e28f-4501-ab13-a4099e4220d3" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640712    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640723    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b8b223-4999-4a21-a275-093515f6ccfe" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640733    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18ce5d6-43d9-4210-a54d-329f8ae1e3d5" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640744    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy"
Oct 30 14:18:27.642582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.640757    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b0d295-010d-4193-8d2d-c9dc02150bd8" containerName="istio-proxy"
Oct 30 14:18:27.649811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.649784    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:27.657389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.657316    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"]
Oct 30 14:18:27.673998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.673970    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b5ea88-openshift-default-dockercfg-4lq6m\""
Oct 30 14:18:27.802836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.802741    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-istio-token\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:27.850911 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.850871    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-envoy\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:27.851405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.851368    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-data\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:27.851630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.851610    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-credential-socket\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:27.851786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.851769    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-socket\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:27.852211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.852057    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77mbj\" (UniqueName: \"kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-kube-api-access-77mbj\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:27.852608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.852586    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-certs\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:27.852760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.852743    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700073be-6e94-45fe-a2e3-d7505c506c85-istio-podinfo\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:27.857776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.852897    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700073be-6e94-45fe-a2e3-d7505c506c85-istiod-ca-cert\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:27.932448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.923926    4530 patch_prober.go:29] interesting pod/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 14:18:27.933327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:27.933286    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 14:18:28.009563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.002283    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-certs\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.009563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.002360    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700073be-6e94-45fe-a2e3-d7505c506c85-istio-podinfo\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.009563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.002409    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700073be-6e94-45fe-a2e3-d7505c506c85-istiod-ca-cert\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.009563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.002462    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-istio-token\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.009563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.002615    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-envoy\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.009563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.002709    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-data\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.009563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.002790    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-credential-socket\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.009563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.002849    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-socket\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.009563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.002911    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77mbj\" (UniqueName: \"kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-kube-api-access-77mbj\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.009563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.004186    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-certs\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.009563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.009108    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-credential-socket\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.009563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.009458    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-data\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.010339 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.010196    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700073be-6e94-45fe-a2e3-d7505c506c85-istiod-ca-cert\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.015558 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.010821    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-socket\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.095613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.090995    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-envoy\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.124570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.114714    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700073be-6e94-45fe-a2e3-d7505c506c85-istio-podinfo\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.150564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.140429    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-istio-token\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.152601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.152566    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77mbj\" (UniqueName: \"kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-kube-api-access-77mbj\") pod \"e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") " pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.232653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.232619    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:18:28.337706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.337676    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:28.593879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.593843    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:18:28.628615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.626865    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:18:28.924027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.922618    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:18:28.924027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.922963    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:18:28.924027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.923020    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:18:28.927189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.927153    4530 patch_prober.go:29] interesting pod/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 14:18:28.927342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.927209    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 14:18:28.930474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:28.930444    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"]
Oct 30 14:18:29.388513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:29.388474    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" event={"ID":"700073be-6e94-45fe-a2e3-d7505c506c85","Type":"ContainerStarted","Data":"72161d94197ddf52c4c8f10520677db3836631b7d791f0374d5430c1aa33aa84"}
Oct 30 14:18:29.388945 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:29.388557    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" event={"ID":"700073be-6e94-45fe-a2e3-d7505c506c85","Type":"ContainerStarted","Data":"5bb87f2043a2e3515891ae70d4268aea198775b9d9d3c1539c58a33c662a2537"}
Oct 30 14:18:29.441065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:29.441013    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" podStartSLOduration=2.441000091 podStartE2EDuration="2.441000091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:29.438528134 +0000 UTC m=+1774.470427072" watchObservedRunningTime="2026-10-30 14:18:29.441000091 +0000 UTC m=+1774.472899029"
Oct 30 14:18:29.919786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:29.919758    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:18:29.921070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:29.921047    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:18:30.307571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.303666    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"]
Oct 30 14:18:30.307571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.304703    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy"
Oct 30 14:18:30.307571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.304721    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2e3f8d-e484-4ec1-9516-d7d34fef7c0a" containerName="istio-proxy"
Oct 30 14:18:30.329102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.325771    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"]
Oct 30 14:18:30.329102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.325890    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.340933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.340155    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:30.346413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.346351    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2b206b-serving-cert\""
Oct 30 14:18:30.350378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.350346    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:30.468769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.468731    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-metrics-certs\") pod \"maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.474615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.474585    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrpm\" (UniqueName: \"kubernetes.io/projected/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-kube-api-access-vjrpm\") pod \"maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.475075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.475055    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-maas-api-tls\") pod \"maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.475264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.475251    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-openshift-service-ca\") pod \"maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.482148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.481953    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:30.483476 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.483458    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:18:30.576761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.576664    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-maas-api-tls\") pod \"maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.576761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.576726    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-openshift-service-ca\") pod \"maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.577031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.576911    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-metrics-certs\") pod \"maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.577100 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.577085    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrpm\" (UniqueName: \"kubernetes.io/projected/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-kube-api-access-vjrpm\") pod \"maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.578396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.578362    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-openshift-service-ca\") pod \"maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.580018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.579991    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-metrics-certs\") pod \"maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.605283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.605253    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-maas-api-tls\") pod \"maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.612483 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.612457    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrpm\" (UniqueName: \"kubernetes.io/projected/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-kube-api-access-vjrpm\") pod \"maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.650936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.650910    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:30.982691 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:18:30.982656    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332016dc_20a2_43cf_9a2e_c2e33c2f2dde.slice/crio-e6bb851a3f543e106b8902802a27d08b7878178b6072a21f0f5e4463e76078cb WatchSource:0}: Error finding container e6bb851a3f543e106b8902802a27d08b7878178b6072a21f0f5e4463e76078cb: Status 404 returned error can't find the container with id e6bb851a3f543e106b8902802a27d08b7878178b6072a21f0f5e4463e76078cb
Oct 30 14:18:30.984512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:30.984490    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"]
Oct 30 14:18:31.492577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:31.492508    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t" event={"ID":"332016dc-20a2-43cf-9a2e-c2e33c2f2dde","Type":"ContainerStarted","Data":"413b8bb71ea0ea5b6a37d31a03f843a49e9e60f90726bb9ad58ff960e9c8466e"}
Oct 30 14:18:31.492980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:31.492622    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:31.492980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:31.492646    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t" event={"ID":"332016dc-20a2-43cf-9a2e-c2e33c2f2dde","Type":"ContainerStarted","Data":"e6bb851a3f543e106b8902802a27d08b7878178b6072a21f0f5e4463e76078cb"}
Oct 30 14:18:31.543075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:31.543028    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t" podStartSLOduration=1.187886297 podStartE2EDuration="1.543015132s" totalImagesPullingTime="355.128835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:31.542163955 +0000 UTC m=+1776.574062893" watchObservedRunningTime="2026-10-30 14:18:31.543015132 +0000 UTC m=+1776.574914070"
Oct 30 14:18:33.383430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.383394    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"]
Oct 30 14:18:33.388115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.388087    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.392840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.392812    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-envoy\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.392984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.392861    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-credential-socket\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.392984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.392963    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt2k6\" (UniqueName: \"kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-kube-api-access-xt2k6\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.393067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.393053    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-data\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.393168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.393152    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-certs\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.393341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.393262    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-token\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.393341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.393318    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-socket\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.393441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.393386    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd82452-192f-4c0c-b16c-0e2522d41778-istiod-ca-cert\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.393441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.393423    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-podinfo\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.412899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.412534    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"]
Oct 30 14:18:33.432603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.421654    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-aab379-openshift-default-dockercfg-p522g\""
Oct 30 14:18:33.511564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.510838    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-data\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.511564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.510966    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-certs\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.511564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.511079    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-token\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.511564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.511134    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-socket\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.511564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.511203    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd82452-192f-4c0c-b16c-0e2522d41778-istiod-ca-cert\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.511564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.511241    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-podinfo\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.511564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.511304    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-envoy\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.511564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.511341    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-credential-socket\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.511564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.511377    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt2k6\" (UniqueName: \"kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-kube-api-access-xt2k6\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.630566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.603649    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd82452-192f-4c0c-b16c-0e2522d41778-istiod-ca-cert\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.630566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.606214    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt2k6\" (UniqueName: \"kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-kube-api-access-xt2k6\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.648624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.645181    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-data\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.661560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.655456    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-certs\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.661560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.655953    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-socket\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.703618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.693483    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-token\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.722594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.721120    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-podinfo\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.734696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.731046    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-credential-socket\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:33.789578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:33.777955    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-envoy\") pod \"e2e-api-b-aab379-openshift-default-69594d6c89-wpdld\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") " pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:34.007894 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.007816    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:34.032916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.032890    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"]
Oct 30 14:18:34.037012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.036990    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"
Oct 30 14:18:34.058695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.058664    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b5ea88\""
Oct 30 14:18:34.058695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.058664    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b5ea88-dockercfg-cmg4m\""
Oct 30 14:18:34.059461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.059436    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"]
Oct 30 14:18:34.151506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.151462    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b00782-6b8b-41d1-badb-a4d9c64b13df-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9\" (UID: \"79b00782-6b8b-41d1-badb-a4d9c64b13df\") " pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"
Oct 30 14:18:34.152229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.152199    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slk5f\" (UniqueName: \"kubernetes.io/projected/79b00782-6b8b-41d1-badb-a4d9c64b13df-kube-api-access-slk5f\") pod \"payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9\" (UID: \"79b00782-6b8b-41d1-badb-a4d9c64b13df\") " pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"
Oct 30 14:18:34.312410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.311842    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slk5f\" (UniqueName: \"kubernetes.io/projected/79b00782-6b8b-41d1-badb-a4d9c64b13df-kube-api-access-slk5f\") pod \"payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9\" (UID: \"79b00782-6b8b-41d1-badb-a4d9c64b13df\") " pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"
Oct 30 14:18:34.312410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.311983    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b00782-6b8b-41d1-badb-a4d9c64b13df-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9\" (UID: \"79b00782-6b8b-41d1-badb-a4d9c64b13df\") " pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"
Oct 30 14:18:34.317290 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.313775    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b00782-6b8b-41d1-badb-a4d9c64b13df-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9\" (UID: \"79b00782-6b8b-41d1-badb-a4d9c64b13df\") " pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"
Oct 30 14:18:34.395572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.387004    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"]
Oct 30 14:18:34.409457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.409112    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"]
Oct 30 14:18:34.409457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.409267    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"
Oct 30 14:18:34.440230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.440184    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slk5f\" (UniqueName: \"kubernetes.io/projected/79b00782-6b8b-41d1-badb-a4d9c64b13df-kube-api-access-slk5f\") pod \"payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9\" (UID: \"79b00782-6b8b-41d1-badb-a4d9c64b13df\") " pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"
Oct 30 14:18:34.518004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.517966    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a617c955-7250-4621-bf43-8ec5419d8434-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v\" (UID: \"a617c955-7250-4621-bf43-8ec5419d8434\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"
Oct 30 14:18:34.518190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.518047    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z44k\" (UniqueName: \"kubernetes.io/projected/a617c955-7250-4621-bf43-8ec5419d8434-kube-api-access-5z44k\") pod \"payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v\" (UID: \"a617c955-7250-4621-bf43-8ec5419d8434\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"
Oct 30 14:18:34.560273 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:18:34.560244    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd82452_192f_4c0c_b16c_0e2522d41778.slice/crio-7b548dfc1f533585f2e901db4e0e8581610ffbcc2b92cd77f9a2616a1463e6ce WatchSource:0}: Error finding container 7b548dfc1f533585f2e901db4e0e8581610ffbcc2b92cd77f9a2616a1463e6ce: Status 404 returned error can't find the container with id 7b548dfc1f533585f2e901db4e0e8581610ffbcc2b92cd77f9a2616a1463e6ce
Oct 30 14:18:34.562986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.562931    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"]
Oct 30 14:18:34.563084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.563020    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:18:34.563124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.563082    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:18:34.563124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.563111    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:18:34.620067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.620034    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z44k\" (UniqueName: \"kubernetes.io/projected/a617c955-7250-4621-bf43-8ec5419d8434-kube-api-access-5z44k\") pod \"payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v\" (UID: \"a617c955-7250-4621-bf43-8ec5419d8434\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"
Oct 30 14:18:34.620385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.620363    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a617c955-7250-4621-bf43-8ec5419d8434-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v\" (UID: \"a617c955-7250-4621-bf43-8ec5419d8434\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"
Oct 30 14:18:34.621283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.621257    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a617c955-7250-4621-bf43-8ec5419d8434-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v\" (UID: \"a617c955-7250-4621-bf43-8ec5419d8434\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"
Oct 30 14:18:34.657417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.657390    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z44k\" (UniqueName: \"kubernetes.io/projected/a617c955-7250-4621-bf43-8ec5419d8434-kube-api-access-5z44k\") pod \"payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v\" (UID: \"a617c955-7250-4621-bf43-8ec5419d8434\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"
Oct 30 14:18:34.660303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.660282    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"
Oct 30 14:18:34.739211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.738124    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" event={"ID":"5dd82452-192f-4c0c-b16c-0e2522d41778","Type":"ContainerStarted","Data":"30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f"}
Oct 30 14:18:34.739211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.738190    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" event={"ID":"5dd82452-192f-4c0c-b16c-0e2522d41778","Type":"ContainerStarted","Data":"7b548dfc1f533585f2e901db4e0e8581610ffbcc2b92cd77f9a2616a1463e6ce"}
Oct 30 14:18:34.767672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.767644    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"
Oct 30 14:18:34.805588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:34.796508    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" podStartSLOduration=1.7964880330000002 podStartE2EDuration="1.796488033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:34.792699501 +0000 UTC m=+1779.824598441" watchObservedRunningTime="2026-10-30 14:18:34.796488033 +0000 UTC m=+1779.828386977"
Oct 30 14:18:35.022566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.017135    4530 patch_prober.go:29] interesting pod/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 14:18:35.022566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.017212    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 14:18:35.022566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.018093    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:35.037067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.037033    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"]
Oct 30 14:18:35.042211 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:18:35.042178    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b00782_6b8b_41d1_badb_a4d9c64b13df.slice/crio-a053b14bf43df7dc6f6f0f4b4d0bd3118c36e31851212cb8ca9f5435c4283416 WatchSource:0}: Error finding container a053b14bf43df7dc6f6f0f4b4d0bd3118c36e31851212cb8ca9f5435c4283416: Status 404 returned error can't find the container with id a053b14bf43df7dc6f6f0f4b4d0bd3118c36e31851212cb8ca9f5435c4283416
Oct 30 14:18:35.140210 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:18:35.140178    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda617c955_7250_4621_bf43_8ec5419d8434.slice/crio-5249c82c92c6228a8b4447e4500f6741e03bc9e91e4d57467d4ddb75f1952815 WatchSource:0}: Error finding container 5249c82c92c6228a8b4447e4500f6741e03bc9e91e4d57467d4ddb75f1952815: Status 404 returned error can't find the container with id 5249c82c92c6228a8b4447e4500f6741e03bc9e91e4d57467d4ddb75f1952815
Oct 30 14:18:35.144041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.144015    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"]
Oct 30 14:18:35.747898 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.747858    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v" event={"ID":"a617c955-7250-4621-bf43-8ec5419d8434","Type":"ContainerStarted","Data":"25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584"}
Oct 30 14:18:35.748381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.747926    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v" event={"ID":"a617c955-7250-4621-bf43-8ec5419d8434","Type":"ContainerStarted","Data":"5249c82c92c6228a8b4447e4500f6741e03bc9e91e4d57467d4ddb75f1952815"}
Oct 30 14:18:35.748381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.747977    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"
Oct 30 14:18:35.750413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.750387    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9" event={"ID":"79b00782-6b8b-41d1-badb-a4d9c64b13df","Type":"ContainerStarted","Data":"f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73"}
Oct 30 14:18:35.750587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.750437    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"
Oct 30 14:18:35.750587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.750461    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9" event={"ID":"79b00782-6b8b-41d1-badb-a4d9c64b13df","Type":"ContainerStarted","Data":"a053b14bf43df7dc6f6f0f4b4d0bd3118c36e31851212cb8ca9f5435c4283416"}
Oct 30 14:18:35.797334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.797288    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v" podStartSLOduration=1.7972749019999998 podStartE2EDuration="1.797274902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:35.793674852 +0000 UTC m=+1780.825573792" watchObservedRunningTime="2026-10-30 14:18:35.797274902 +0000 UTC m=+1780.829173841"
Oct 30 14:18:35.843654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:35.843612    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9" podStartSLOduration=1.843600798 podStartE2EDuration="1.843600798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:35.840507788 +0000 UTC m=+1780.872406727" watchObservedRunningTime="2026-10-30 14:18:35.843600798 +0000 UTC m=+1780.875499736"
Oct 30 14:18:36.022862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:36.020722    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:36.763698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:36.763663    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:36.768381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:36.768355    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:18:37.555751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:37.555718    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:18:38.985585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:38.985551    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"]
Oct 30 14:18:38.989053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:38.989036    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"
Oct 30 14:18:39.011890 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.011864    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"]
Oct 30 14:18:39.020261 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.020198    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-aab379\""
Oct 30 14:18:39.020261 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.020198    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-aab379-dockercfg-nwb9n\""
Oct 30 14:18:39.117927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.117881    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c82gm\" (UniqueName: \"kubernetes.io/projected/94883728-37a4-412f-9330-01fd7d181855-kube-api-access-c82gm\") pod \"payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9\" (UID: \"94883728-37a4-412f-9330-01fd7d181855\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"
Oct 30 14:18:39.118256 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.118230    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94883728-37a4-412f-9330-01fd7d181855-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9\" (UID: \"94883728-37a4-412f-9330-01fd7d181855\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"
Oct 30 14:18:39.226568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.221024    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c82gm\" (UniqueName: \"kubernetes.io/projected/94883728-37a4-412f-9330-01fd7d181855-kube-api-access-c82gm\") pod \"payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9\" (UID: \"94883728-37a4-412f-9330-01fd7d181855\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"
Oct 30 14:18:39.226568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.221305    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94883728-37a4-412f-9330-01fd7d181855-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9\" (UID: \"94883728-37a4-412f-9330-01fd7d181855\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"
Oct 30 14:18:39.254612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.246163    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94883728-37a4-412f-9330-01fd7d181855-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9\" (UID: \"94883728-37a4-412f-9330-01fd7d181855\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"
Oct 30 14:18:39.291328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.291289    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"]
Oct 30 14:18:39.315986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.315951    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"]
Oct 30 14:18:39.316336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.316273    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"
Oct 30 14:18:39.352937 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.343144    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c82gm\" (UniqueName: \"kubernetes.io/projected/94883728-37a4-412f-9330-01fd7d181855-kube-api-access-c82gm\") pod \"payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9\" (UID: \"94883728-37a4-412f-9330-01fd7d181855\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"
Oct 30 14:18:39.437581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.430276    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtfkf\" (UniqueName: \"kubernetes.io/projected/cb7a5a16-b15e-4724-88c2-853c833cd9dc-kube-api-access-jtfkf\") pod \"payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl\" (UID: \"cb7a5a16-b15e-4724-88c2-853c833cd9dc\") " pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"
Oct 30 14:18:39.437581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.430528    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7a5a16-b15e-4724-88c2-853c833cd9dc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl\" (UID: \"cb7a5a16-b15e-4724-88c2-853c833cd9dc\") " pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"
Oct 30 14:18:39.532210 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.532126    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtfkf\" (UniqueName: \"kubernetes.io/projected/cb7a5a16-b15e-4724-88c2-853c833cd9dc-kube-api-access-jtfkf\") pod \"payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl\" (UID: \"cb7a5a16-b15e-4724-88c2-853c833cd9dc\") " pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"
Oct 30 14:18:39.532432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.532269    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7a5a16-b15e-4724-88c2-853c833cd9dc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl\" (UID: \"cb7a5a16-b15e-4724-88c2-853c833cd9dc\") " pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"
Oct 30 14:18:39.533050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.533023    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7a5a16-b15e-4724-88c2-853c833cd9dc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl\" (UID: \"cb7a5a16-b15e-4724-88c2-853c833cd9dc\") " pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"
Oct 30 14:18:39.559250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.559224    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtfkf\" (UniqueName: \"kubernetes.io/projected/cb7a5a16-b15e-4724-88c2-853c833cd9dc-kube-api-access-jtfkf\") pod \"payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl\" (UID: \"cb7a5a16-b15e-4724-88c2-853c833cd9dc\") " pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"
Oct 30 14:18:39.603946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.603922    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"
Oct 30 14:18:39.722634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.722607    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"
Oct 30 14:18:39.908548 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:18:39.908497    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94883728_37a4_412f_9330_01fd7d181855.slice/crio-e351d6f60a44259961829d3ca4168a537c53c57862ec30b566e2f1b7f228a98c WatchSource:0}: Error finding container e351d6f60a44259961829d3ca4168a537c53c57862ec30b566e2f1b7f228a98c: Status 404 returned error can't find the container with id e351d6f60a44259961829d3ca4168a537c53c57862ec30b566e2f1b7f228a98c
Oct 30 14:18:39.910673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:39.910648    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"]
Oct 30 14:18:40.051291 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:18:40.051224    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7a5a16_b15e_4724_88c2_853c833cd9dc.slice/crio-5c09c4d736a9b4aca8cdfcc9ea45b9015dea5538ae880290cb7a977947488d89 WatchSource:0}: Error finding container 5c09c4d736a9b4aca8cdfcc9ea45b9015dea5538ae880290cb7a977947488d89: Status 404 returned error can't find the container with id 5c09c4d736a9b4aca8cdfcc9ea45b9015dea5538ae880290cb7a977947488d89
Oct 30 14:18:40.054771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:40.054748    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"]
Oct 30 14:18:40.805500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:40.805463    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl" event={"ID":"cb7a5a16-b15e-4724-88c2-853c833cd9dc","Type":"ContainerStarted","Data":"70a36e7d57a52984c11b26c13e5c40c3bb72b651da37b9427cfdb7205c6d80f1"}
Oct 30 14:18:40.805730 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:40.805516    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl" event={"ID":"cb7a5a16-b15e-4724-88c2-853c833cd9dc","Type":"ContainerStarted","Data":"5c09c4d736a9b4aca8cdfcc9ea45b9015dea5538ae880290cb7a977947488d89"}
Oct 30 14:18:40.805730 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:40.805674    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"
Oct 30 14:18:40.807329 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:40.807307    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9" event={"ID":"94883728-37a4-412f-9330-01fd7d181855","Type":"ContainerStarted","Data":"75130e8bee7f11637b4d4964832b7cca1cf1ba449c990d13251ba8a1c0e4fde9"}
Oct 30 14:18:40.807438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:40.807343    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9" event={"ID":"94883728-37a4-412f-9330-01fd7d181855","Type":"ContainerStarted","Data":"e351d6f60a44259961829d3ca4168a537c53c57862ec30b566e2f1b7f228a98c"}
Oct 30 14:18:40.807438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:40.807376    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"
Oct 30 14:18:40.856574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:40.856519    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl" podStartSLOduration=1.856508874 podStartE2EDuration="1.856508874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:40.852741604 +0000 UTC m=+1785.884640546" watchObservedRunningTime="2026-10-30 14:18:40.856508874 +0000 UTC m=+1785.888407824"
Oct 30 14:18:40.911584 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:40.911531    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9" podStartSLOduration=2.911520224 podStartE2EDuration="2.911520224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:40.908132578 +0000 UTC m=+1785.940031528" watchObservedRunningTime="2026-10-30 14:18:40.911520224 +0000 UTC m=+1785.943419162"
Oct 30 14:18:41.368700 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:18:41.368665    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:18:43.167864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.162770    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"]
Oct 30 14:18:43.169316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.169289    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"
Oct 30 14:18:43.195451 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.195426    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9dba4af9\""
Oct 30 14:18:43.219588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.217893    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"]
Oct 30 14:18:43.227205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.227175    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"]
Oct 30 14:18:43.231818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.231795    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"
Oct 30 14:18:43.260220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.260193    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9dba4af9-dockercfg-jfjxp\""
Oct 30 14:18:43.260460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.260432    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"]
Oct 30 14:18:43.300436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.300397    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fafa95-69f0-4942-a5e4-1843e3beb361-plugins-config-volume\") pod \"payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d\" (UID: \"67fafa95-69f0-4942-a5e4-1843e3beb361\") " pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"
Oct 30 14:18:43.300923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.300518    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhckv\" (UniqueName: \"kubernetes.io/projected/67fafa95-69f0-4942-a5e4-1843e3beb361-kube-api-access-vhckv\") pod \"payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d\" (UID: \"67fafa95-69f0-4942-a5e4-1843e3beb361\") " pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"
Oct 30 14:18:43.300923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.300751    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a34205-1dd8-44eb-89cb-e89a8f3c553e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw\" (UID: \"44a34205-1dd8-44eb-89cb-e89a8f3c553e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"
Oct 30 14:18:43.300923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.300808    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvb6f\" (UniqueName: \"kubernetes.io/projected/44a34205-1dd8-44eb-89cb-e89a8f3c553e-kube-api-access-gvb6f\") pod \"payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw\" (UID: \"44a34205-1dd8-44eb-89cb-e89a8f3c553e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"
Oct 30 14:18:43.423619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.414892    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fafa95-69f0-4942-a5e4-1843e3beb361-plugins-config-volume\") pod \"payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d\" (UID: \"67fafa95-69f0-4942-a5e4-1843e3beb361\") " pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"
Oct 30 14:18:43.424069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.424024    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhckv\" (UniqueName: \"kubernetes.io/projected/67fafa95-69f0-4942-a5e4-1843e3beb361-kube-api-access-vhckv\") pod \"payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d\" (UID: \"67fafa95-69f0-4942-a5e4-1843e3beb361\") " pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"
Oct 30 14:18:43.424468 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.424442    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a34205-1dd8-44eb-89cb-e89a8f3c553e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw\" (UID: \"44a34205-1dd8-44eb-89cb-e89a8f3c553e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"
Oct 30 14:18:43.424671 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.424651    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvb6f\" (UniqueName: \"kubernetes.io/projected/44a34205-1dd8-44eb-89cb-e89a8f3c553e-kube-api-access-gvb6f\") pod \"payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw\" (UID: \"44a34205-1dd8-44eb-89cb-e89a8f3c553e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"
Oct 30 14:18:43.449615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.426854    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fafa95-69f0-4942-a5e4-1843e3beb361-plugins-config-volume\") pod \"payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d\" (UID: \"67fafa95-69f0-4942-a5e4-1843e3beb361\") " pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"
Oct 30 14:18:43.449867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.436333    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a34205-1dd8-44eb-89cb-e89a8f3c553e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw\" (UID: \"44a34205-1dd8-44eb-89cb-e89a8f3c553e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"
Oct 30 14:18:43.474394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.474362    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvb6f\" (UniqueName: \"kubernetes.io/projected/44a34205-1dd8-44eb-89cb-e89a8f3c553e-kube-api-access-gvb6f\") pod \"payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw\" (UID: \"44a34205-1dd8-44eb-89cb-e89a8f3c553e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"
Oct 30 14:18:43.486624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.482280    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhckv\" (UniqueName: \"kubernetes.io/projected/67fafa95-69f0-4942-a5e4-1843e3beb361-kube-api-access-vhckv\") pod \"payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d\" (UID: \"67fafa95-69f0-4942-a5e4-1843e3beb361\") " pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"
Oct 30 14:18:43.486624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.486021    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"
Oct 30 14:18:43.552375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:43.551784    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"
Oct 30 14:18:43.999472 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:18:43.999445    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fafa95_69f0_4942_a5e4_1843e3beb361.slice/crio-3d545a250b06a07af5b475784309aab3a783f887697f9c6a79728de13f1dddc6 WatchSource:0}: Error finding container 3d545a250b06a07af5b475784309aab3a783f887697f9c6a79728de13f1dddc6: Status 404 returned error can't find the container with id 3d545a250b06a07af5b475784309aab3a783f887697f9c6a79728de13f1dddc6
Oct 30 14:18:44.002759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:44.002734    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"]
Oct 30 14:18:44.031531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:44.031507    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"]
Oct 30 14:18:44.034837 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:18:44.034812    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a34205_1dd8_44eb_89cb_e89a8f3c553e.slice/crio-5cbf8da0186bfbd5bc24cc0d9247cec0a7efd368394f41045698034bc91bf2e5 WatchSource:0}: Error finding container 5cbf8da0186bfbd5bc24cc0d9247cec0a7efd368394f41045698034bc91bf2e5: Status 404 returned error can't find the container with id 5cbf8da0186bfbd5bc24cc0d9247cec0a7efd368394f41045698034bc91bf2e5
Oct 30 14:18:44.846462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:44.846427    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw" event={"ID":"44a34205-1dd8-44eb-89cb-e89a8f3c553e","Type":"ContainerStarted","Data":"c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664"}
Oct 30 14:18:44.846987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:44.846490    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw" event={"ID":"44a34205-1dd8-44eb-89cb-e89a8f3c553e","Type":"ContainerStarted","Data":"5cbf8da0186bfbd5bc24cc0d9247cec0a7efd368394f41045698034bc91bf2e5"}
Oct 30 14:18:44.846987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:44.846519    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"
Oct 30 14:18:44.848520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:44.848491    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d" event={"ID":"67fafa95-69f0-4942-a5e4-1843e3beb361","Type":"ContainerStarted","Data":"fad847147c2a759d184f3b9d80e9b91fde6e62e20c80b73bc61864bf5db21976"}
Oct 30 14:18:44.848634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:44.848588    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"
Oct 30 14:18:44.848634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:44.848619    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d" event={"ID":"67fafa95-69f0-4942-a5e4-1843e3beb361","Type":"ContainerStarted","Data":"3d545a250b06a07af5b475784309aab3a783f887697f9c6a79728de13f1dddc6"}
Oct 30 14:18:44.895458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:44.895412    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw" podStartSLOduration=1.895398921 podStartE2EDuration="1.895398921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:44.891789048 +0000 UTC m=+1789.923687987" watchObservedRunningTime="2026-10-30 14:18:44.895398921 +0000 UTC m=+1789.927297860"
Oct 30 14:18:44.940718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:44.940667    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d" podStartSLOduration=1.940654741 podStartE2EDuration="1.940654741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:44.936626723 +0000 UTC m=+1789.968525662" watchObservedRunningTime="2026-10-30 14:18:44.940654741 +0000 UTC m=+1789.972553681"
Oct 30 14:18:46.180992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:46.180960    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:18:46.765063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:46.765030    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"
Oct 30 14:18:46.765672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:46.765650    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"
Oct 30 14:18:50.095566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:50.092472    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:18:50.437740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:50.437635    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:18:50.799627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:50.792643    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr" podUID="91d82203-dd8e-4e1f-ad76-0403aea6eaca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:18:50.799627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:50.793101    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 14:18:51.837371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:51.837339    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"
Oct 30 14:18:51.837876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:51.837666    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"
Oct 30 14:18:52.044632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:52.044602    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"]
Oct 30 14:18:52.052536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:52.052502    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40d77425-xksls"]
Oct 30 14:18:53.137870 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:53.137837    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025875c5-a966-4f05-bf7f-b68c9ba00b73" path="/var/lib/kubelet/pods/025875c5-a966-4f05-bf7f-b68c9ba00b73/volumes"
Oct 30 14:18:54.546791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.546758    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"]
Oct 30 14:18:54.551131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.551111    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.597698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.588682    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"]
Oct 30 14:18:54.597698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.588735    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9dba4af9-serving-cert\""
Oct 30 14:18:54.635582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.634252    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rxx7\" (UniqueName: \"kubernetes.io/projected/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-kube-api-access-5rxx7\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.635582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.634447    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-metrics-certs\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.635582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.634509    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-openshift-service-ca\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.635582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.634573    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-maas-api-tls\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.736778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.736717    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-metrics-certs\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.736981 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.736822    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-openshift-service-ca\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.736981 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.736870    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-maas-api-tls\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.737152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.737085    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rxx7\" (UniqueName: \"kubernetes.io/projected/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-kube-api-access-5rxx7\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.738107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.738076    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-openshift-service-ca\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.740027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.739999    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-metrics-certs\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.740501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.740478    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-maas-api-tls\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.849613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.840127    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rxx7\" (UniqueName: \"kubernetes.io/projected/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-kube-api-access-5rxx7\") pod \"maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:54.877838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:54.877804    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:55.678638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:55.678528    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:18:55.710313 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:55.710281    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:18:55.714364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:55.714346    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:18:55.715889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:55.715869    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:18:55.716177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:55.716158    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:18:55.716923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:55.716905    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:18:55.810572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:55.810509    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"]
Oct 30 14:18:55.868844 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:55.868808    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"
Oct 30 14:18:55.868991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:55.868922    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"
Oct 30 14:18:56.652335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:56.652234    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw" event={"ID":"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b","Type":"ContainerStarted","Data":"7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a"}
Oct 30 14:18:56.652335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:56.652327    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw" event={"ID":"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b","Type":"ContainerStarted","Data":"bef0c3cf32403fde1088d5ebde187edc124f34450ae5fce4ee1529b72e9763e7"}
Oct 30 14:18:56.652594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:56.652350    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:18:56.702219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:18:56.702171    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw" podStartSLOduration=2.170052104 podStartE2EDuration="2.702157936s" totalImagesPullingTime="532.105832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:56.701637012 +0000 UTC m=+1801.733535952" watchObservedRunningTime="2026-10-30 14:18:56.702157936 +0000 UTC m=+1801.734056875"
Oct 30 14:19:00.570391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.570353    4530 patch_prober.go:29] interesting pod/e2e-api-a-d74808-openshift-default-5454f78567-g69q7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 14:19:00.570927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.570413    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 14:19:00.806982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.806922    4530 generic.go:356] "Generic (PLEG): container finished" podID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerID="5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e" exitCode=0
Oct 30 14:19:00.807349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.807222    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" event={"ID":"75a4813f-02bc-4368-be3c-dcb62d5273cf","Type":"ContainerDied","Data":"5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e"}
Oct 30 14:19:00.811713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.811197    4530 scope.go:122] "RemoveContainer" containerID="5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e"
Oct 30 14:19:00.816439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.816396    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:00.816620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.816483    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:00.816620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.816522    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:00.816620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.816536    4530 generic.go:356] "Generic (PLEG): container finished" podID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerID="9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2" exitCode=0
Oct 30 14:19:00.816620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.816604    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" event={"ID":"8ace7927-a93c-4a75-879d-be6e0b8aac79","Type":"ContainerDied","Data":"9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2"}
Oct 30 14:19:00.817160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.817088    4530 scope.go:122] "RemoveContainer" containerID="9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2"
Oct 30 14:19:00.819012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.818983    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:00.819124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.819062    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:00.819124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:00.819113    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:01.008475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:01.006875    4530 scope.go:122] "RemoveContainer" containerID="5d21aee4955b2a767aeed92b1978856d9de894a2edfbc713858b14c7c35003f7"
Oct 30 14:19:01.022191 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:01.022169    4530 scope.go:122] "RemoveContainer" containerID="6f4be218a3b043515fa90c2e4543428bfb8410e8e04fadeccd943166492fff92"
Oct 30 14:19:01.034171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:01.034075    4530 scope.go:122] "RemoveContainer" containerID="4ea30a449e45e08031e679d67679dbe84c32765bb320afb613fc5837446bf682"
Oct 30 14:19:01.061669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:01.061639    4530 scope.go:122] "RemoveContainer" containerID="fe6e9085b1022d1ca06e87d3bdd23d9d92f6dfb9abf126a8618d05d0a8244f50"
Oct 30 14:19:01.085708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:01.085687    4530 scope.go:122] "RemoveContainer" containerID="3df0cc180fcfd71b61859a3efa544d7514d57c71536b1dc4e4be4b317c0d30c5"
Oct 30 14:19:01.097949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:01.097861    4530 scope.go:122] "RemoveContainer" containerID="1327422b92d18b6227c6dc79a41ad16289707209a807c97eacc12f0b826bb1fb"
Oct 30 14:19:01.107984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:01.107960    4530 scope.go:122] "RemoveContainer" containerID="0620aac666c48b1fbbc957c859fb6521a35db2bb3cdaa61c7b7e1116d59aa670"
Oct 30 14:19:01.845618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:01.845576    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" event={"ID":"75a4813f-02bc-4368-be3c-dcb62d5273cf","Type":"ContainerStarted","Data":"5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330"}
Oct 30 14:19:01.847920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:01.847888    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" event={"ID":"8ace7927-a93c-4a75-879d-be6e0b8aac79","Type":"ContainerStarted","Data":"38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b"}
Oct 30 14:19:01.901564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:01.891942    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:19:01.901564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:01.892062    4530 patch_prober.go:29] interesting pod/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 14:19:01.901564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:01.892121    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 14:19:02.216624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:02.215960    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:19:02.216624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:02.216098    4530 patch_prober.go:29] interesting pod/e2e-api-a-d74808-openshift-default-5454f78567-g69q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 14:19:02.216624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:02.216148    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 14:19:02.668316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:02.668276    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:19:02.918621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:02.917874    4530 patch_prober.go:29] interesting pod/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 14:19:02.940185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:02.917936    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 14:19:03.219040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:03.218948    4530 patch_prober.go:29] interesting pod/e2e-api-a-d74808-openshift-default-5454f78567-g69q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 14:19:03.219299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:03.219269    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 14:19:03.903567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:03.899908    4530 patch_prober.go:29] interesting pod/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 14:19:03.903567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:03.899971    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 14:19:04.209419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:04.208912    4530 patch_prober.go:29] interesting pod/e2e-api-a-d74808-openshift-default-5454f78567-g69q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 14:19:04.209419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:04.208974    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 14:19:04.895566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:04.892045    4530 patch_prober.go:29] interesting pod/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 14:19:04.895566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:04.892105    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 14:19:05.216623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:05.208930    4530 patch_prober.go:29] interesting pod/e2e-api-a-d74808-openshift-default-5454f78567-g69q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 14:19:05.216623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:05.209007    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 14:19:05.934601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:05.928323    4530 patch_prober.go:29] interesting pod/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 14:19:05.934601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:05.928388    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 14:19:06.209012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:06.208916    4530 patch_prober.go:29] interesting pod/e2e-api-a-d74808-openshift-default-5454f78567-g69q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 14:19:06.209265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:06.209237    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 14:19:06.892922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:06.892884    4530 patch_prober.go:29] interesting pod/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 14:19:06.893563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:06.893465    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 14:19:07.258362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:07.258269    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:19:07.258985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:07.258958    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:19:07.259941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:07.259919    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:19:07.896735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:07.896705    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:19:07.897243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:07.897212    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:19:07.898511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:07.898488    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:19:09.769289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.769253    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"]
Oct 30 14:19:09.776597 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.776571    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:09.783171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.783145    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"]
Oct 30 14:19:09.803593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.801512    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2a2835-openshift-default-dockercfg-s5hmr\""
Oct 30 14:19:09.923562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.920005    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a45514-45c1-441e-ad52-b48b6072eac8-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:09.923562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.920119    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:09.923562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.920322    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:09.923562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.920503    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:09.923562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.920609    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a45514-45c1-441e-ad52-b48b6072eac8-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:09.923562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.920865    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:09.923562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.920914    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp8sp\" (UniqueName: \"kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-kube-api-access-rp8sp\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:09.923562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.920943    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:09.923562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:09.921003    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.024051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.023850    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.024051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.023915    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.024051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.023994    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a45514-45c1-441e-ad52-b48b6072eac8-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.024341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.024156    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.024341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.024182    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8sp\" (UniqueName: \"kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-kube-api-access-rp8sp\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.024341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.024212    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.024341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.024264    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.024341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.024301    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a45514-45c1-441e-ad52-b48b6072eac8-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.024646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.024382    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.037561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.025323    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.037561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.026843    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.037561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.027089    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.037561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.027326    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.037561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.028132    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a45514-45c1-441e-ad52-b48b6072eac8-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.041559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.040726    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a45514-45c1-441e-ad52-b48b6072eac8-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.069562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.060349    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8sp\" (UniqueName: \"kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-kube-api-access-rp8sp\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.069562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.064869    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.073563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.072377    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.150573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.146979    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:10.622767 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:19:10.622740    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a45514_45c1_441e_ad52_b48b6072eac8.slice/crio-029d9b11e040c9af0e4641418f798e9dc97cd9d9a76c91fb81a50861843632ab WatchSource:0}: Error finding container 029d9b11e040c9af0e4641418f798e9dc97cd9d9a76c91fb81a50861843632ab: Status 404 returned error can't find the container with id 029d9b11e040c9af0e4641418f798e9dc97cd9d9a76c91fb81a50861843632ab
Oct 30 14:19:10.625889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.625854    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:10.625997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.625908    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"]
Oct 30 14:19:10.625997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.625923    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:10.625997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:10.625963    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:11.180431 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:11.180396    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" event={"ID":"72a45514-45c1-441e-ad52-b48b6072eac8","Type":"ContainerStarted","Data":"b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94"}
Oct 30 14:19:11.180852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:11.180447    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" event={"ID":"72a45514-45c1-441e-ad52-b48b6072eac8","Type":"ContainerStarted","Data":"029d9b11e040c9af0e4641418f798e9dc97cd9d9a76c91fb81a50861843632ab"}
Oct 30 14:19:11.247971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:11.247913    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" podStartSLOduration=2.247899443 podStartE2EDuration="2.247899443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:11.243798824 +0000 UTC m=+1816.275697787" watchObservedRunningTime="2026-10-30 14:19:11.247899443 +0000 UTC m=+1816.279798381"
Oct 30 14:19:11.478417 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:19:11.478325    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:19:11.711487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:11.711447    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"]
Oct 30 14:19:11.711761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:11.711730    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy" containerID="cri-o://95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b" gracePeriod=30
Oct 30 14:19:11.716505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:11.716480    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"]
Oct 30 14:19:11.716722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:11.716700    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" podUID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" containerName="istio-proxy" containerID="cri-o://d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f" gracePeriod=30
Oct 30 14:19:11.725143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:11.725117    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"]
Oct 30 14:19:11.725295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:11.725277    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy" containerID="cri-o://c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3" gracePeriod=30
Oct 30 14:19:12.147964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:12.147930    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:12.154350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:12.154316    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:12.188428 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:12.188400    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:12.189354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:12.189330    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:19:14.189866 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.189829    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"]
Oct 30 14:19:14.193631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.193596    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.226263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.226238    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b5ea88-serving-cert\""
Oct 30 14:19:14.235078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.235051    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"]
Oct 30 14:19:14.262788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.262748    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-maas-api-tls\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.263084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.263058    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdvh8\" (UniqueName: \"kubernetes.io/projected/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-kube-api-access-kdvh8\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.263260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.263233    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-openshift-service-ca\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.263742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.263516    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-metrics-certs\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.365030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.364999    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-metrics-certs\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.365224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.365116    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-maas-api-tls\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.365224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.365147    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdvh8\" (UniqueName: \"kubernetes.io/projected/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-kube-api-access-kdvh8\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.365224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.365169    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-openshift-service-ca\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.365490 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:19:14.365254    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-serving-cert: secret "maas-api-e2e-api-b-b5ea88-serving-cert" not found
Oct 30 14:19:14.365490 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:19:14.365331    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-maas-api-tls podName:d1ef98d5-ba45-41aa-bbdf-4dca262731f9 nodeName:}" failed. No retries permitted until 2026-10-30 14:19:14.865309257 +0000 UTC m=+1819.897208194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-maas-api-tls") pod "maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx" (UID: "d1ef98d5-ba45-41aa-bbdf-4dca262731f9") : secret "maas-api-e2e-api-b-b5ea88-serving-cert" not found
Oct 30 14:19:14.366016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.365989    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-openshift-service-ca\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.367592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.367571    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-metrics-certs\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.407237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.403470    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdvh8\" (UniqueName: \"kubernetes.io/projected/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-kube-api-access-kdvh8\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.879036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.879004    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-maas-api-tls\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:14.881410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:14.881389    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-maas-api-tls\") pod \"maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:15.111899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:15.111863    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:15.420824 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:19:15.420758    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ef98d5_ba45_41aa_bbdf_4dca262731f9.slice/crio-11b020ec902c0f33045489c8210650f3df51884ac7c252e90886a1b2d56c684d WatchSource:0}: Error finding container 11b020ec902c0f33045489c8210650f3df51884ac7c252e90886a1b2d56c684d: Status 404 returned error can't find the container with id 11b020ec902c0f33045489c8210650f3df51884ac7c252e90886a1b2d56c684d
Oct 30 14:19:15.422584 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:15.422560    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"]
Oct 30 14:19:16.228934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:16.228846    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx" event={"ID":"d1ef98d5-ba45-41aa-bbdf-4dca262731f9","Type":"ContainerStarted","Data":"685dd24c875ceaa5faee1bfcd2b7657f1dda049da71c166355ff2825f347b9f2"}
Oct 30 14:19:16.228934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:16.228912    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx" event={"ID":"d1ef98d5-ba45-41aa-bbdf-4dca262731f9","Type":"ContainerStarted","Data":"11b020ec902c0f33045489c8210650f3df51884ac7c252e90886a1b2d56c684d"}
Oct 30 14:19:16.229112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:16.228939    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:16.281432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:16.281381    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx" podStartSLOduration=1.765936296 podStartE2EDuration="2.281366659s" totalImagesPullingTime="515.430363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:16.277185339 +0000 UTC m=+1821.309084279" watchObservedRunningTime="2026-10-30 14:19:16.281366659 +0000 UTC m=+1821.313265644"
Oct 30 14:19:17.035221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.035191    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:19:17.054755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.054729    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:19:17.059079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.059057    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:19:17.134524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.134492    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-token\" (UniqueName: \"kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-token\") pod \"ec037157-c68e-45fe-ba80-7cf66725d2bf\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") "
Oct 30 14:19:17.134802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.134581    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-credential-socket\") pod \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") "
Oct 30 14:19:17.134802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.134614    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-socket\") pod \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") "
Oct 30 14:19:17.134802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.134642    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-kube-api-access-rvkbb\" (UniqueName: \"kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-kube-api-access-rvkbb\") pod \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") "
Oct 30 14:19:17.134802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.134681    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istiod-ca-cert\") pod \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") "
Oct 30 14:19:17.134802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.134722    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec037157-c68e-45fe-ba80-7cf66725d2bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec037157-c68e-45fe-ba80-7cf66725d2bf-istiod-ca-cert\") pod \"ec037157-c68e-45fe-ba80-7cf66725d2bf\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") "
Oct 30 14:19:17.134802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.134768    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-token\" (UniqueName: \"kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-token\") pod \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") "
Oct 30 14:19:17.135165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.134814    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-envoy\") pod \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") "
Oct 30 14:19:17.135165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.134893    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-envoy\") pod \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") "
Oct 30 14:19:17.135165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.134907    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-socket" pod "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" (UID: "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.135165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.134932    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-certs\") pod \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") "
Oct 30 14:19:17.135165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.134986    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-data\") pod \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") "
Oct 30 14:19:17.135165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135018    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-certs\") pod \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") "
Oct 30 14:19:17.135165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135128    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-socket\") pod \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") "
Oct 30 14:19:17.135165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135133    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istiod-ca-cert" pod "bb4c0803-2f0b-4718-aca6-a4b43ae47c20" (UID: "bb4c0803-2f0b-4718-aca6-a4b43ae47c20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:19:17.135165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135136    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec037157-c68e-45fe-ba80-7cf66725d2bf-istiod-ca-cert" pod "ec037157-c68e-45fe-ba80-7cf66725d2bf" (UID: "ec037157-c68e-45fe-ba80-7cf66725d2bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:19:17.135165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135161    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-envoy\") pod \"ec037157-c68e-45fe-ba80-7cf66725d2bf\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") "
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135198    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-credential-socket\") pod \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") "
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135271    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-kube-api-access-n6qhx\" (UniqueName: \"kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-kube-api-access-n6qhx\") pod \"ec037157-c68e-45fe-ba80-7cf66725d2bf\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") "
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135318    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-socket\") pod \"ec037157-c68e-45fe-ba80-7cf66725d2bf\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") "
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135356    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-kube-api-access-t6rrt\" (UniqueName: \"kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-kube-api-access-t6rrt\") pod \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") "
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135416    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-podinfo\") pod \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") "
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135472    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-credential-socket\") pod \"ec037157-c68e-45fe-ba80-7cf66725d2bf\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") "
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135507    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-podinfo\") pod \"ec037157-c68e-45fe-ba80-7cf66725d2bf\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") "
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135556    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istiod-ca-cert\") pod \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") "
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135575    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-credential-socket" pod "bb4c0803-2f0b-4718-aca6-a4b43ae47c20" (UID: "bb4c0803-2f0b-4718-aca6-a4b43ae47c20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135582    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-token\" (UniqueName: \"kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-token\") pod \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") "
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135647    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-certs" pod "bb4c0803-2f0b-4718-aca6-a4b43ae47c20" (UID: "bb4c0803-2f0b-4718-aca6-a4b43ae47c20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135658    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-data\") pod \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\" (UID: \"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c\") "
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135773    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-certs" pod "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" (UID: "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.135784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135777    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-data" pod "bb4c0803-2f0b-4718-aca6-a4b43ae47c20" (UID: "bb4c0803-2f0b-4718-aca6-a4b43ae47c20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135993    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-socket" pod "ec037157-c68e-45fe-ba80-7cf66725d2bf" (UID: "ec037157-c68e-45fe-ba80-7cf66725d2bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.135983    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-data\") pod \"ec037157-c68e-45fe-ba80-7cf66725d2bf\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") "
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.136066    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-podinfo\") pod \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\" (UID: \"bb4c0803-2f0b-4718-aca6-a4b43ae47c20\") "
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.136158    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-certs\") pod \"ec037157-c68e-45fe-ba80-7cf66725d2bf\" (UID: \"ec037157-c68e-45fe-ba80-7cf66725d2bf\") "
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.136180    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-socket" pod "bb4c0803-2f0b-4718-aca6-a4b43ae47c20" (UID: "bb4c0803-2f0b-4718-aca6-a4b43ae47c20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.136187    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-data" pod "ec037157-c68e-45fe-ba80-7cf66725d2bf" (UID: "ec037157-c68e-45fe-ba80-7cf66725d2bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.136526    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-credential-socket" pod "ec037157-c68e-45fe-ba80-7cf66725d2bf" (UID: "ec037157-c68e-45fe-ba80-7cf66725d2bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.136740    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-certs" pod "ec037157-c68e-45fe-ba80-7cf66725d2bf" (UID: "ec037157-c68e-45fe-ba80-7cf66725d2bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.137032    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.137057    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.137071    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.137084    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.137101    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.137117    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.137133    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.137148    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.137161    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.137174    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.137189    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.137727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.137203    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec037157-c68e-45fe-ba80-7cf66725d2bf-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.139739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.138354    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-credential-socket" pod "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" (UID: "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.139739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.138408    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-data" pod "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" (UID: "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.139739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.138935    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-token" pod "ec037157-c68e-45fe-ba80-7cf66725d2bf" (UID: "ec037157-c68e-45fe-ba80-7cf66725d2bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:19:17.139739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.139301    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istiod-ca-cert" pod "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" (UID: "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:19:17.140968 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.140940    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-kube-api-access-t6rrt" pod "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" (UID: "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c"). InnerVolumeSpecName "kube-api-access-t6rrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:19:17.141229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.141175    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-envoy" pod "ec037157-c68e-45fe-ba80-7cf66725d2bf" (UID: "ec037157-c68e-45fe-ba80-7cf66725d2bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.141355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.141223    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-kube-api-access-n6qhx" pod "ec037157-c68e-45fe-ba80-7cf66725d2bf" (UID: "ec037157-c68e-45fe-ba80-7cf66725d2bf"). InnerVolumeSpecName "kube-api-access-n6qhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:19:17.141355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.141277    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-podinfo" pod "bb4c0803-2f0b-4718-aca6-a4b43ae47c20" (UID: "bb4c0803-2f0b-4718-aca6-a4b43ae47c20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:19:17.141666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.141622    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-envoy" pod "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" (UID: "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.141666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.141627    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-token" pod "bb4c0803-2f0b-4718-aca6-a4b43ae47c20" (UID: "bb4c0803-2f0b-4718-aca6-a4b43ae47c20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:19:17.141836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.141812    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-kube-api-access-rvkbb" pod "bb4c0803-2f0b-4718-aca6-a4b43ae47c20" (UID: "bb4c0803-2f0b-4718-aca6-a4b43ae47c20"). InnerVolumeSpecName "kube-api-access-rvkbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:19:17.142749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.142629    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-envoy" pod "bb4c0803-2f0b-4718-aca6-a4b43ae47c20" (UID: "bb4c0803-2f0b-4718-aca6-a4b43ae47c20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:17.142749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.142709    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-podinfo" pod "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" (UID: "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:19:17.142749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.142726    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-token" pod "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" (UID: "d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:19:17.143452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.143427    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-podinfo" pod "ec037157-c68e-45fe-ba80-7cf66725d2bf" (UID: "ec037157-c68e-45fe-ba80-7cf66725d2bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238592    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238628    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238642    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238656    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238669    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238683    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6qhx\" (UniqueName: \"kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-kube-api-access-n6qhx\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238696    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6rrt\" (UniqueName: \"kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-kube-api-access-t6rrt\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238710    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238723    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238737    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238749    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238761    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238775    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238787    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec037157-c68e-45fe-ba80-7cf66725d2bf-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.239021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.238800    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvkbb\" (UniqueName: \"kubernetes.io/projected/bb4c0803-2f0b-4718-aca6-a4b43ae47c20-kube-api-access-rvkbb\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:17.243335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.243299    4530 generic.go:356] "Generic (PLEG): container finished" podID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" containerID="d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f" exitCode=0
Oct 30 14:19:17.243648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.243596    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" event={"ID":"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c","Type":"ContainerDied","Data":"d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f"}
Oct 30 14:19:17.243722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.243658    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc" event={"ID":"d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c","Type":"ContainerDied","Data":"afdf03e9562ab6d2a9e1ae7e4780f7b41f968baac9680b2aa9cca58cb851ded1"}
Oct 30 14:19:17.243722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.243690    4530 scope.go:122] "RemoveContainer" containerID="d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f"
Oct 30 14:19:17.243872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.243855    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"
Oct 30 14:19:17.245960 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.245935    4530 generic.go:356] "Generic (PLEG): container finished" podID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerID="c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3" exitCode=0
Oct 30 14:19:17.246068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.246044    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" event={"ID":"bb4c0803-2f0b-4718-aca6-a4b43ae47c20","Type":"ContainerDied","Data":"c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3"}
Oct 30 14:19:17.246152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.246088    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t" event={"ID":"bb4c0803-2f0b-4718-aca6-a4b43ae47c20","Type":"ContainerDied","Data":"ee0ce7be2a2c00dde4c4ed8f847b1576d2143b3d94a1d4f346f471fcde4031da"}
Oct 30 14:19:17.246215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.246157    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"
Oct 30 14:19:17.253918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.253893    4530 generic.go:356] "Generic (PLEG): container finished" podID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerID="95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b" exitCode=0
Oct 30 14:19:17.254024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.253928    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"
Oct 30 14:19:17.254024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.253962    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" event={"ID":"ec037157-c68e-45fe-ba80-7cf66725d2bf","Type":"ContainerDied","Data":"95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b"}
Oct 30 14:19:17.254024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.254003    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4" event={"ID":"ec037157-c68e-45fe-ba80-7cf66725d2bf","Type":"ContainerDied","Data":"bff480bd09663c7c62681e8e75829ea11e2f6644fa6cbd28448757959716cbdf"}
Oct 30 14:19:17.258143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.258123    4530 scope.go:122] "RemoveContainer" containerID="d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f"
Oct 30 14:19:17.258474 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:19:17.258450    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f\": container with ID starting with d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f not found: ID does not exist" containerID="d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f"
Oct 30 14:19:17.258622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.258478    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f"} err="failed to get container status \"d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f\": rpc error: code = NotFound desc = could not find container \"d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f\": container with ID starting with d48daa6776ae6680d10751bc95c55b22d9b9eb9ad4c2e65ec037457bec9e805f not found: ID does not exist"
Oct 30 14:19:17.258622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.258494    4530 scope.go:122] "RemoveContainer" containerID="c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3"
Oct 30 14:19:17.271388 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.271366    4530 scope.go:122] "RemoveContainer" containerID="c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3"
Oct 30 14:19:17.271669 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:19:17.271639    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3\": container with ID starting with c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3 not found: ID does not exist" containerID="c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3"
Oct 30 14:19:17.271756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.271668    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3"} err="failed to get container status \"c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3\": rpc error: code = NotFound desc = could not find container \"c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3\": container with ID starting with c35d33d86b3b7dfce60019ffb57c4a980139cd6b45f591fe8655daeb9cad30b3 not found: ID does not exist"
Oct 30 14:19:17.271756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.271693    4530 scope.go:122] "RemoveContainer" containerID="95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b"
Oct 30 14:19:17.281667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.281646    4530 scope.go:122] "RemoveContainer" containerID="95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b"
Oct 30 14:19:17.281924 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:19:17.281906    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b\": container with ID starting with 95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b not found: ID does not exist" containerID="95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b"
Oct 30 14:19:17.281975 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.281930    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b"} err="failed to get container status \"95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b\": rpc error: code = NotFound desc = could not find container \"95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b\": container with ID starting with 95b42fe077e742409bc4ba01476e4476b08bca7c12499ac7feade7eee714bc1b not found: ID does not exist"
Oct 30 14:19:17.310112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.310090    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"]
Oct 30 14:19:17.314771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.314738    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-cftgc"]
Oct 30 14:19:17.403301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.403261    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"]
Oct 30 14:19:17.410211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.410183    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-jch2t"]
Oct 30 14:19:17.456811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.456779    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"]
Oct 30 14:19:17.470448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:17.470414    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-wsnc4"]
Oct 30 14:19:18.278818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.278788    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"]
Oct 30 14:19:18.279737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.279686    4530 cpu_manager.go:414] "containerMap: removing container" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy"
Oct 30 14:19:18.279737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.279733    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy"
Oct 30 14:19:18.279948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.279817    4530 cpu_manager.go:414] "containerMap: removing container" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy"
Oct 30 14:19:18.279948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.279824    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy"
Oct 30 14:19:18.279948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.279836    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" containerName="istio-proxy"
Oct 30 14:19:18.279948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.279842    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" containerName="istio-proxy"
Oct 30 14:19:18.280175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.279961    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" containerName="istio-proxy"
Oct 30 14:19:18.280175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.279975    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" containerName="istio-proxy"
Oct 30 14:19:18.280175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.279981    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" containerName="istio-proxy"
Oct 30 14:19:18.284357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.284332    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.309510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.309485    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1c4b82-openshift-default-dockercfg-fskpj\""
Oct 30 14:19:18.319799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.319769    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"]
Oct 30 14:19:18.379881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.379849    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-podinfo\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.380021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.379899    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-certs\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.380021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.379917    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xchg8\" (UniqueName: \"kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-kube-api-access-xchg8\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.380120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.380049    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-credential-socket\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.380120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.380081    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-socket\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.380188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.380137    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-data\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.380220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.380187    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-token\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.380220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.380212    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-envoy\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.380285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.380254    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istiod-ca-cert\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.486552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.485825    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-podinfo\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.486552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.485906    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-certs\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.486552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.485930    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xchg8\" (UniqueName: \"kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-kube-api-access-xchg8\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.486552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.485967    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-credential-socket\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.486552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.485995    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-socket\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.486552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.486016    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-data\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.486552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.486046    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-token\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.486552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.486072    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-envoy\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.486552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.486121    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istiod-ca-cert\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.487330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.487062    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istiod-ca-cert\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.495562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.487762    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-socket\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.495562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.488187    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-certs\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.511566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.506225    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-data\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.532611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.518875    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-credential-socket\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.532611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.520285    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-podinfo\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.533147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.533115    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-envoy\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.543563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.537382    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xchg8\" (UniqueName: \"kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-kube-api-access-xchg8\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.562166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.562130    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-token\") pod \"e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") " pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:18.604070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:18.603667    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:19.081036 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:19:19.081008    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8c9ae2_bbe3_4d19_8360_3728e5ffa25a.slice/crio-58132235ac7ee15a738ce518edbcd7e12ccdd79970d7280caca660da6842f6f6 WatchSource:0}: Error finding container 58132235ac7ee15a738ce518edbcd7e12ccdd79970d7280caca660da6842f6f6: Status 404 returned error can't find the container with id 58132235ac7ee15a738ce518edbcd7e12ccdd79970d7280caca660da6842f6f6
Oct 30 14:19:19.083434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:19.083404    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"]
Oct 30 14:19:19.084423 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:19.084386    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:19.084530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:19.084472    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:19.084530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:19.084521    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:19.117913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:19.117885    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4c0803-2f0b-4718-aca6-a4b43ae47c20" path="/var/lib/kubelet/pods/bb4c0803-2f0b-4718-aca6-a4b43ae47c20/volumes"
Oct 30 14:19:19.118464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:19.118443    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c" path="/var/lib/kubelet/pods/d5b7b8d4-21ed-441c-8a3a-92936fe9ab4c/volumes"
Oct 30 14:19:19.118948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:19.118932    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec037157-c68e-45fe-ba80-7cf66725d2bf" path="/var/lib/kubelet/pods/ec037157-c68e-45fe-ba80-7cf66725d2bf/volumes"
Oct 30 14:19:19.276480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:19.276448    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" event={"ID":"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a","Type":"ContainerStarted","Data":"9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22"}
Oct 30 14:19:19.276648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:19.276497    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" event={"ID":"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a","Type":"ContainerStarted","Data":"58132235ac7ee15a738ce518edbcd7e12ccdd79970d7280caca660da6842f6f6"}
Oct 30 14:19:19.363750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:19.363645    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" podStartSLOduration=1.36362674 podStartE2EDuration="1.36362674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:19.355330308 +0000 UTC m=+1824.387229248" watchObservedRunningTime="2026-10-30 14:19:19.36362674 +0000 UTC m=+1824.395525680"
Oct 30 14:19:19.608982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:19.608942    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:20.299000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.298961    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"]
Oct 30 14:19:20.381566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.374399    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"]
Oct 30 14:19:20.381566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.374522    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.398574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.393827    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-aab379-serving-cert\""
Oct 30 14:19:20.628520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.618692    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-metrics-certs\") pod \"maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.628520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.618742    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eb2d85-8e94-49a4-b651-2aef801a540b-openshift-service-ca\") pod \"maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.628520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.618848    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz7gx\" (UniqueName: \"kubernetes.io/projected/16eb2d85-8e94-49a4-b651-2aef801a540b-kube-api-access-bz7gx\") pod \"maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.628520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.619032    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-maas-api-tls\") pod \"maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.635787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.635757    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:20.720244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.720208    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-maas-api-tls\") pod \"maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.720408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.720388    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-metrics-certs\") pod \"maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.720473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.720424    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eb2d85-8e94-49a4-b651-2aef801a540b-openshift-service-ca\") pod \"maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.720528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.720511    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz7gx\" (UniqueName: \"kubernetes.io/projected/16eb2d85-8e94-49a4-b651-2aef801a540b-kube-api-access-bz7gx\") pod \"maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.721207 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.721183    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eb2d85-8e94-49a4-b651-2aef801a540b-openshift-service-ca\") pod \"maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.722792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.722770    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-maas-api-tls\") pod \"maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.722880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.722840    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-metrics-certs\") pod \"maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.755205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.755178    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz7gx\" (UniqueName: \"kubernetes.io/projected/16eb2d85-8e94-49a4-b651-2aef801a540b-kube-api-access-bz7gx\") pod \"maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:20.885130 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:20.885039    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:21.197465 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:19:21.197435    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16eb2d85_8e94_49a4_b651_2aef801a540b.slice/crio-3002a0eceaedd81b5fc6b07f939c37d3256bdb23412d80b93e59c383bd6a9a97 WatchSource:0}: Error finding container 3002a0eceaedd81b5fc6b07f939c37d3256bdb23412d80b93e59c383bd6a9a97: Status 404 returned error can't find the container with id 3002a0eceaedd81b5fc6b07f939c37d3256bdb23412d80b93e59c383bd6a9a97
Oct 30 14:19:21.198478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:21.198455    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"]
Oct 30 14:19:21.420653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:21.420611    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5" event={"ID":"16eb2d85-8e94-49a4-b651-2aef801a540b","Type":"ContainerStarted","Data":"3002a0eceaedd81b5fc6b07f939c37d3256bdb23412d80b93e59c383bd6a9a97"}
Oct 30 14:19:21.421061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:21.420916    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:21.422071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:21.422043    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:19:22.280567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:22.280106    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:19:22.480561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:22.474363    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5" event={"ID":"16eb2d85-8e94-49a4-b651-2aef801a540b","Type":"ContainerStarted","Data":"b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd"}
Oct 30 14:19:22.480561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:22.474443    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:22.555662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:22.553509    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5" podStartSLOduration=2.003925908 podStartE2EDuration="2.553487898s" totalImagesPullingTime="549.56199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:22.552113883 +0000 UTC m=+1827.584012849" watchObservedRunningTime="2026-10-30 14:19:22.553487898 +0000 UTC m=+1827.585386839"
Oct 30 14:19:29.636915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:29.636886    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:19:36.657226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:36.657187    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"]
Oct 30 14:19:36.657892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:36.657432    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy" containerID="cri-o://be5ea1c5f317588d73daae9a84f347098a3f02e0123de4e0880bfe6d7aaaaa65" gracePeriod=30
Oct 30 14:19:41.371447 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:19:41.371360    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:19:41.906343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:41.906302    4530 generic.go:356] "Generic (PLEG): container finished" podID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerID="be5ea1c5f317588d73daae9a84f347098a3f02e0123de4e0880bfe6d7aaaaa65" exitCode=0
Oct 30 14:19:41.906576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:41.906380    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" event={"ID":"42dedd41-345f-4dd6-8519-fa7587c4a383","Type":"ContainerDied","Data":"be5ea1c5f317588d73daae9a84f347098a3f02e0123de4e0880bfe6d7aaaaa65"}
Oct 30 14:19:41.906576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:41.906456    4530 scope.go:122] "RemoveContainer" containerID="e475cb9e369d7116ad6ddac404d7a1a5ca626df82a91b67afe04079a0d2fd7ae"
Oct 30 14:19:41.968668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:41.968644    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:19:42.086416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086382    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-certs\") pod \"42dedd41-345f-4dd6-8519-fa7587c4a383\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") "
Oct 30 14:19:42.086630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086468    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42dedd41-345f-4dd6-8519-fa7587c4a383-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42dedd41-345f-4dd6-8519-fa7587c4a383-istiod-ca-cert\") pod \"42dedd41-345f-4dd6-8519-fa7587c4a383\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") "
Oct 30 14:19:42.086727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086628    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-data\") pod \"42dedd41-345f-4dd6-8519-fa7587c4a383\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") "
Oct 30 14:19:42.086727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086657    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-certs" pod "42dedd41-345f-4dd6-8519-fa7587c4a383" (UID: "42dedd41-345f-4dd6-8519-fa7587c4a383"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:42.086727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086690    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-credential-socket\") pod \"42dedd41-345f-4dd6-8519-fa7587c4a383\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") "
Oct 30 14:19:42.086727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086720    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-envoy\") pod \"42dedd41-345f-4dd6-8519-fa7587c4a383\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") "
Oct 30 14:19:42.086942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086738    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-kube-api-access-tz2l5\" (UniqueName: \"kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-kube-api-access-tz2l5\") pod \"42dedd41-345f-4dd6-8519-fa7587c4a383\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") "
Oct 30 14:19:42.086942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086774    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-podinfo\") pod \"42dedd41-345f-4dd6-8519-fa7587c4a383\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") "
Oct 30 14:19:42.086942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086857    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-token\" (UniqueName: \"kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-token\") pod \"42dedd41-345f-4dd6-8519-fa7587c4a383\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") "
Oct 30 14:19:42.086942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086879    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-data" pod "42dedd41-345f-4dd6-8519-fa7587c4a383" (UID: "42dedd41-345f-4dd6-8519-fa7587c4a383"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:42.086942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086891    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-socket\") pod \"42dedd41-345f-4dd6-8519-fa7587c4a383\" (UID: \"42dedd41-345f-4dd6-8519-fa7587c4a383\") "
Oct 30 14:19:42.086942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086932    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-credential-socket" pod "42dedd41-345f-4dd6-8519-fa7587c4a383" (UID: "42dedd41-345f-4dd6-8519-fa7587c4a383"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:42.087260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.086976    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42dedd41-345f-4dd6-8519-fa7587c4a383-istiod-ca-cert" pod "42dedd41-345f-4dd6-8519-fa7587c4a383" (UID: "42dedd41-345f-4dd6-8519-fa7587c4a383"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:19:42.087260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.087225    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-socket" pod "42dedd41-345f-4dd6-8519-fa7587c4a383" (UID: "42dedd41-345f-4dd6-8519-fa7587c4a383"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:42.087665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.087640    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:42.087773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.087669    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:42.087773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.087683    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:42.087773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.087704    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:42.087773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.087720    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42dedd41-345f-4dd6-8519-fa7587c4a383-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:42.089067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.089038    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-token" pod "42dedd41-345f-4dd6-8519-fa7587c4a383" (UID: "42dedd41-345f-4dd6-8519-fa7587c4a383"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:19:42.089178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.089095    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-podinfo" pod "42dedd41-345f-4dd6-8519-fa7587c4a383" (UID: "42dedd41-345f-4dd6-8519-fa7587c4a383"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:19:42.089178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.089117    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-envoy" pod "42dedd41-345f-4dd6-8519-fa7587c4a383" (UID: "42dedd41-345f-4dd6-8519-fa7587c4a383"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:19:42.089309 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.089196    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-kube-api-access-tz2l5" pod "42dedd41-345f-4dd6-8519-fa7587c4a383" (UID: "42dedd41-345f-4dd6-8519-fa7587c4a383"). InnerVolumeSpecName "kube-api-access-tz2l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:19:42.188647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.188535    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:42.188647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.188609    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz2l5\" (UniqueName: \"kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-kube-api-access-tz2l5\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:42.188647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.188621    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:42.188647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.188630    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42dedd41-345f-4dd6-8519-fa7587c4a383-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:19:42.915713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.915680    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"
Oct 30 14:19:42.916168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.915716    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc" event={"ID":"42dedd41-345f-4dd6-8519-fa7587c4a383","Type":"ContainerDied","Data":"af6d291b18c805407214d77957e02abab0b440e8d09cd261e886c2fa9bb79555"}
Oct 30 14:19:42.916168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.915773    4530 scope.go:122] "RemoveContainer" containerID="be5ea1c5f317588d73daae9a84f347098a3f02e0123de4e0880bfe6d7aaaaa65"
Oct 30 14:19:42.993963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:42.993937    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"]
Oct 30 14:19:43.007611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:43.007588    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-01955278-openshift-default-56c64d6797-9hfnc"]
Oct 30 14:19:43.116236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:43.116211    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" path="/var/lib/kubelet/pods/42dedd41-345f-4dd6-8519-fa7587c4a383/volumes"
Oct 30 14:19:44.300564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:44.294375    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:19:46.278736 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.278681    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"]
Oct 30 14:19:46.279488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.279468    4530 cpu_manager.go:414] "containerMap: removing container" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy"
Oct 30 14:19:46.279488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.279488    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy"
Oct 30 14:19:46.279689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.279503    4530 cpu_manager.go:414] "containerMap: removing container" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy"
Oct 30 14:19:46.279689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.279508    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy"
Oct 30 14:19:46.279689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.279660    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy"
Oct 30 14:19:46.279689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.279670    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="42dedd41-345f-4dd6-8519-fa7587c4a383" containerName="istio-proxy"
Oct 30 14:19:46.285243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.285222    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.305246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.305217    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"]
Oct 30 14:19:46.313740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.313719    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5a9d7b-openshift-default-dockercfg-k58x9\""
Oct 30 14:19:46.423108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.423074    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-credential-socket\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.423272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.423144    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-data\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.423272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.423251    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z97m\" (UniqueName: \"kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-kube-api-access-5z97m\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.423348 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.423282    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-envoy\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.423348 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.423300    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-podinfo\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.423422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.423352    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-socket\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.423422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.423409    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-certs\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.423582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.423568    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc89bfd-e9d1-4821-883c-18f959c7793f-istiod-ca-cert\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.423626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.423599    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-token\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.528360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.526344    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-data\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.528360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.526447    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z97m\" (UniqueName: \"kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-kube-api-access-5z97m\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.528360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.526475    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-envoy\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.528360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.526502    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-podinfo\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.528360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.526558    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-socket\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.528360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.526601    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-certs\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.528360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.526770    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc89bfd-e9d1-4821-883c-18f959c7793f-istiod-ca-cert\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.528360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.526826    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-token\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.528360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.526936    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-credential-socket\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.528360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.527446    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-credential-socket\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.528360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.527731    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-data\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.536830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.536322    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-certs\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.549742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.549706    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc89bfd-e9d1-4821-883c-18f959c7793f-istiod-ca-cert\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.550411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.550385    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-socket\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.570612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.570525    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z97m\" (UniqueName: \"kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-kube-api-access-5z97m\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.585500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.585456    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-token\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.607665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.607364    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-podinfo\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.607905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.607570    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-envoy\") pod \"e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") " pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:46.910936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:46.910892    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:47.240734 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:19:47.240693    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc89bfd_e9d1_4821_883c_18f959c7793f.slice/crio-7249e92d9ab31735131385b775a63238d623884795e3cfd2c8ebabeb8a49d2ab WatchSource:0}: Error finding container 7249e92d9ab31735131385b775a63238d623884795e3cfd2c8ebabeb8a49d2ab: Status 404 returned error can't find the container with id 7249e92d9ab31735131385b775a63238d623884795e3cfd2c8ebabeb8a49d2ab
Oct 30 14:19:47.242027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:47.241999    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"]
Oct 30 14:19:47.243824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:47.243788    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:47.243927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:47.243858    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:47.243927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:47.243887    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:47.962243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:47.960605    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"]
Oct 30 14:19:47.966682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:47.966655    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.038124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.038092    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4bf33a-openshift-default-dockercfg-vzf88\""
Oct 30 14:19:48.046494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.041335    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"]
Oct 30 14:19:48.063981 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.063944    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" event={"ID":"cdc89bfd-e9d1-4821-883c-18f959c7793f","Type":"ContainerStarted","Data":"029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9"}
Oct 30 14:19:48.064256 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.064232    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" event={"ID":"cdc89bfd-e9d1-4821-883c-18f959c7793f","Type":"ContainerStarted","Data":"7249e92d9ab31735131385b775a63238d623884795e3cfd2c8ebabeb8a49d2ab"}
Oct 30 14:19:48.067741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.067692    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-data\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.068068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.068046    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istiod-ca-cert\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.068267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.068254    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-token\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.068430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.068414    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-certs\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.068647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.068631    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-socket\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.068789 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.068776    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-credential-socket\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.068965 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.068952    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-podinfo\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.069108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.069094    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nfck\" (UniqueName: \"kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-kube-api-access-9nfck\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.069235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.069222    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-envoy\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.177603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.173408    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-data\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.196665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.196635    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istiod-ca-cert\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.198349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.198319    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-token\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.198605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.197891    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istiod-ca-cert\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.216613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.195522    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-data\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.217482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.216856    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-certs\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.217674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.217239    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-certs\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.263052 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.262999    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-socket\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.263362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.263339    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-credential-socket\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.305179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.305142    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-podinfo\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.305428 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.305401    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nfck\" (UniqueName: \"kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-kube-api-access-9nfck\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.305620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.305601    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-envoy\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.305860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.305842    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-credential-socket\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.306013 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.265660    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-socket\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.386570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.383141    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-envoy\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.430579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.427459    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nfck\" (UniqueName: \"kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-kube-api-access-9nfck\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.468631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.445120    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" podStartSLOduration=2.4451010269999998 podStartE2EDuration="2.445101027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:48.264258137 +0000 UTC m=+1853.296157077" watchObservedRunningTime="2026-10-30 14:19:48.445101027 +0000 UTC m=+1853.476999969"
Oct 30 14:19:48.468631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.467162    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-token\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.468631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.468401    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-podinfo\") pod \"e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") " pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.643579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.639123    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:48.913260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.913230    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:48.917683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:48.917657    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:49.031933 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:19:49.031902    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d7564b_c0c0_4031_b7e9_d77bfe2b7472.slice/crio-e5d050853d711df24bdeb140769ee1caa196ab0318f66b9e928b77ad2faca7c5 WatchSource:0}: Error finding container e5d050853d711df24bdeb140769ee1caa196ab0318f66b9e928b77ad2faca7c5: Status 404 returned error can't find the container with id e5d050853d711df24bdeb140769ee1caa196ab0318f66b9e928b77ad2faca7c5
Oct 30 14:19:49.034245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:49.034220    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"]
Oct 30 14:19:49.035424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:49.035381    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:49.035517 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:49.035471    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:49.035583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:49.035516    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:49.073887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:49.073862    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" event={"ID":"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472","Type":"ContainerStarted","Data":"e5d050853d711df24bdeb140769ee1caa196ab0318f66b9e928b77ad2faca7c5"}
Oct 30 14:19:49.074170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:49.074152    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:49.075215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:49.075198    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:19:50.106459 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:50.106420    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" event={"ID":"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472","Type":"ContainerStarted","Data":"7a86d1bebd2b06b31b251cda9d9c22c43c4533959a6f0a34b5a6df2a866bbb43"}
Oct 30 14:19:50.203564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:50.200677    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" podStartSLOduration=3.200659417 podStartE2EDuration="3.200659417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:50.169676224 +0000 UTC m=+1855.201575173" watchObservedRunningTime="2026-10-30 14:19:50.200659417 +0000 UTC m=+1855.232558358"
Oct 30 14:19:50.639743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:50.639709    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:50.645667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:50.645641    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:51.119530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:51.119496    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:51.119967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:51.119802    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:19:56.001385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:56.001349    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9591-f7a415"]
Oct 30 14:19:56.007400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:56.007376    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-f7a415"
Oct 30 14:19:56.023902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:56.023866    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9591-f7a415"]
Oct 30 14:19:56.029361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:56.029336    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q95l7\""
Oct 30 14:19:56.073382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:56.073351    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mrm5\" (UniqueName: \"kubernetes.io/projected/614a2f09-c9ff-4fea-af01-b28e41f1c3ec-kube-api-access-7mrm5\") pod \"test-curl-9591-f7a415\" (UID: \"614a2f09-c9ff-4fea-af01-b28e41f1c3ec\") " pod="opendatahub/test-curl-9591-f7a415"
Oct 30 14:19:56.174921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:56.174889    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mrm5\" (UniqueName: \"kubernetes.io/projected/614a2f09-c9ff-4fea-af01-b28e41f1c3ec-kube-api-access-7mrm5\") pod \"test-curl-9591-f7a415\" (UID: \"614a2f09-c9ff-4fea-af01-b28e41f1c3ec\") " pod="opendatahub/test-curl-9591-f7a415"
Oct 30 14:19:56.209089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:56.208917    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mrm5\" (UniqueName: \"kubernetes.io/projected/614a2f09-c9ff-4fea-af01-b28e41f1c3ec-kube-api-access-7mrm5\") pod \"test-curl-9591-f7a415\" (UID: \"614a2f09-c9ff-4fea-af01-b28e41f1c3ec\") " pod="opendatahub/test-curl-9591-f7a415"
Oct 30 14:19:56.324071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:56.324037    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-f7a415"
Oct 30 14:19:56.634507 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:19:56.634479    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod614a2f09_c9ff_4fea_af01_b28e41f1c3ec.slice/crio-24fa0d63d55911c55171692da49ebf7d8bb5de608d4ebb28c25b41dd6c382b9b WatchSource:0}: Error finding container 24fa0d63d55911c55171692da49ebf7d8bb5de608d4ebb28c25b41dd6c382b9b: Status 404 returned error can't find the container with id 24fa0d63d55911c55171692da49ebf7d8bb5de608d4ebb28c25b41dd6c382b9b
Oct 30 14:19:56.638562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:56.638509    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9591-f7a415"]
Oct 30 14:19:57.315383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:57.315349    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9591-f7a415" event={"ID":"614a2f09-c9ff-4fea-af01-b28e41f1c3ec","Type":"ContainerStarted","Data":"2943d6c7c325c2da8a831379529ac443c381cb11e19eadc12a7ae404078fa4f6"}
Oct 30 14:19:57.315383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:57.315389    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9591-f7a415" event={"ID":"614a2f09-c9ff-4fea-af01-b28e41f1c3ec","Type":"ContainerStarted","Data":"24fa0d63d55911c55171692da49ebf7d8bb5de608d4ebb28c25b41dd6c382b9b"}
Oct 30 14:19:57.364460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:57.364407    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9591-f7a415" podStartSLOduration=2.364389885 podStartE2EDuration="2.364389885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:57.358759071 +0000 UTC m=+1862.390658012" watchObservedRunningTime="2026-10-30 14:19:57.364389885 +0000 UTC m=+1862.396289022"
Oct 30 14:19:58.272064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:58.272026    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9591-f7a415"]
Oct 30 14:19:58.278488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:58.278454    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9591-f7a415"]
Oct 30 14:19:58.325130 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:58.325100    4530 generic.go:356] "Generic (PLEG): container finished" podID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerID="30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f" exitCode=0
Oct 30 14:19:58.325633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:58.325176    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" event={"ID":"5dd82452-192f-4c0c-b16c-0e2522d41778","Type":"ContainerDied","Data":"30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f"}
Oct 30 14:19:58.325788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:58.325771    4530 scope.go:122] "RemoveContainer" containerID="30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f"
Oct 30 14:19:58.327403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:58.327368    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:58.327486 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:58.327439    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:58.327486 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:58.327471    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:19:59.335362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:59.335322    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" event={"ID":"5dd82452-192f-4c0c-b16c-0e2522d41778","Type":"ContainerStarted","Data":"7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be"}
Oct 30 14:19:59.335843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:19:59.335674    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9591-f7a415" podUID="614a2f09-c9ff-4fea-af01-b28e41f1c3ec" containerName="test-curl-9591-f7a415" containerID="cri-o://2943d6c7c325c2da8a831379529ac443c381cb11e19eadc12a7ae404078fa4f6" gracePeriod=2
Oct 30 14:20:00.008449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:00.008409    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:20:00.008822 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:00.008794    4530 patch_prober.go:29] interesting pod/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 14:20:00.008952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:00.008852    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 14:20:01.008316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:01.008286    4530 patch_prober.go:29] interesting pod/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 14:20:01.008735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:01.008337    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 14:20:01.277193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:01.277109    4530 scope.go:122] "RemoveContainer" containerID="d0e82a0271c8c53e2cc1664c41fc56d77fd580b945a5d9dcc2ed3b74dd7d570b"
Oct 30 14:20:01.314617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:01.314592    4530 scope.go:122] "RemoveContainer" containerID="6274f79be39d43d60abb3c2917d881baeb61192be1fdd2a9930960d60e50857c"
Oct 30 14:20:01.324307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:01.324289    4530 scope.go:122] "RemoveContainer" containerID="b6ae0d4bc6e27062b3f46b7c43e0b9c43c062a024bd51d4ae92fd7398323909e"
Oct 30 14:20:01.333644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:01.333625    4530 scope.go:122] "RemoveContainer" containerID="b5edf4dcb7f1e1dc41fbcd63bd6060abc7ee010609699d284f14b3b10a7e5eb7"
Oct 30 14:20:01.343222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:01.343199    4530 scope.go:122] "RemoveContainer" containerID="838f41ea41840ca566f9c48501d2cebc892f3ccc6fa4cbe7ea1e20cf61d89510"
Oct 30 14:20:01.875567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:01.870709    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-f7a415"
Oct 30 14:20:01.994626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:01.994112    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614a2f09-c9ff-4fea-af01-b28e41f1c3ec-kube-api-access-7mrm5\" (UniqueName: \"kubernetes.io/projected/614a2f09-c9ff-4fea-af01-b28e41f1c3ec-kube-api-access-7mrm5\") pod \"614a2f09-c9ff-4fea-af01-b28e41f1c3ec\" (UID: \"614a2f09-c9ff-4fea-af01-b28e41f1c3ec\") "
Oct 30 14:20:02.013589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:02.009110    4530 patch_prober.go:29] interesting pod/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 14:20:02.013589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:02.009172    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 14:20:02.013589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:02.011279    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614a2f09-c9ff-4fea-af01-b28e41f1c3ec-kube-api-access-7mrm5" pod "614a2f09-c9ff-4fea-af01-b28e41f1c3ec" (UID: "614a2f09-c9ff-4fea-af01-b28e41f1c3ec"). InnerVolumeSpecName "kube-api-access-7mrm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:20:02.097568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:02.097210    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mrm5\" (UniqueName: \"kubernetes.io/projected/614a2f09-c9ff-4fea-af01-b28e41f1c3ec-kube-api-access-7mrm5\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:02.368752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:02.368720    4530 generic.go:356] "Generic (PLEG): container finished" podID="614a2f09-c9ff-4fea-af01-b28e41f1c3ec" containerID="2943d6c7c325c2da8a831379529ac443c381cb11e19eadc12a7ae404078fa4f6" exitCode=137
Oct 30 14:20:02.368954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:02.368761    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-f7a415"
Oct 30 14:20:02.368954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:02.368825    4530 scope.go:122] "RemoveContainer" containerID="2943d6c7c325c2da8a831379529ac443c381cb11e19eadc12a7ae404078fa4f6"
Oct 30 14:20:02.400572 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:20:02.399781    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9591-f7a415\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="614a2f09-c9ff-4fea-af01-b28e41f1c3ec" pod="opendatahub/test-curl-9591-f7a415"
Oct 30 14:20:02.418568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:02.416875    4530 scope.go:122] "RemoveContainer" containerID="2943d6c7c325c2da8a831379529ac443c381cb11e19eadc12a7ae404078fa4f6"
Oct 30 14:20:02.418568 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:20:02.417307    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2943d6c7c325c2da8a831379529ac443c381cb11e19eadc12a7ae404078fa4f6\": container with ID starting with 2943d6c7c325c2da8a831379529ac443c381cb11e19eadc12a7ae404078fa4f6 not found: ID does not exist" containerID="2943d6c7c325c2da8a831379529ac443c381cb11e19eadc12a7ae404078fa4f6"
Oct 30 14:20:02.418568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:02.417341    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2943d6c7c325c2da8a831379529ac443c381cb11e19eadc12a7ae404078fa4f6"} err="failed to get container status \"2943d6c7c325c2da8a831379529ac443c381cb11e19eadc12a7ae404078fa4f6\": rpc error: code = NotFound desc = could not find container \"2943d6c7c325c2da8a831379529ac443c381cb11e19eadc12a7ae404078fa4f6\": container with ID starting with 2943d6c7c325c2da8a831379529ac443c381cb11e19eadc12a7ae404078fa4f6 not found: ID does not exist"
Oct 30 14:20:03.024577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:03.023204    4530 patch_prober.go:29] interesting pod/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 14:20:03.024577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:03.023281    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 14:20:03.138906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:03.138876    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614a2f09-c9ff-4fea-af01-b28e41f1c3ec" path="/var/lib/kubelet/pods/614a2f09-c9ff-4fea-af01-b28e41f1c3ec/volumes"
Oct 30 14:20:04.014565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:04.011038    4530 patch_prober.go:29] interesting pod/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 14:20:04.014565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:04.011099    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 14:20:05.171854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:05.171133    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:20:05.173075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:05.172684    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:20:05.174330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:05.174303    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:20:07.895612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:07.890390    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"]
Oct 30 14:20:07.895612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:07.891676    4530 cpu_manager.go:414] "containerMap: removing container" podUID="614a2f09-c9ff-4fea-af01-b28e41f1c3ec" containerName="test-curl-9591-f7a415"
Oct 30 14:20:07.895612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:07.891699    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614a2f09-c9ff-4fea-af01-b28e41f1c3ec" containerName="test-curl-9591-f7a415"
Oct 30 14:20:07.895612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:07.891856    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="614a2f09-c9ff-4fea-af01-b28e41f1c3ec" containerName="test-curl-9591-f7a415"
Oct 30 14:20:07.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:07.912409    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:07.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:07.928787    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"]
Oct 30 14:20:07.951587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:07.937269    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1c4b82-serving-cert\""
Oct 30 14:20:07.996569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:07.996106    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxz6\" (UniqueName: \"kubernetes.io/projected/3a91d81b-7562-4463-95ce-b35e8d65a664-kube-api-access-frxz6\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:07.996569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:07.996230    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a91d81b-7562-4463-95ce-b35e8d65a664-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:07.996569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:07.996323    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-metrics-certs\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:07.996569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:07.996394    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:08.098574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:08.098094    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:08.098574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:08.098325    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxz6\" (UniqueName: \"kubernetes.io/projected/3a91d81b-7562-4463-95ce-b35e8d65a664-kube-api-access-frxz6\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:08.098574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:08.098398    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a91d81b-7562-4463-95ce-b35e8d65a664-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:08.098574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:08.098477    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-metrics-certs\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:08.103560 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:20:08.099667    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-serving-cert: secret "maas-api-e2e-api-b-1c4b82-serving-cert" not found
Oct 30 14:20:08.103560 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:20:08.099749    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-maas-api-tls podName:3a91d81b-7562-4463-95ce-b35e8d65a664 nodeName:}" failed. No retries permitted until 2026-10-30 14:20:08.599731638 +0000 UTC m=+1873.631630560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-maas-api-tls") pod "maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s" (UID: "3a91d81b-7562-4463-95ce-b35e8d65a664") : secret "maas-api-e2e-api-b-1c4b82-serving-cert" not found
Oct 30 14:20:08.103560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:08.100015    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a91d81b-7562-4463-95ce-b35e8d65a664-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:08.103560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:08.102236    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-metrics-certs\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:08.131986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:08.127877    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxz6\" (UniqueName: \"kubernetes.io/projected/3a91d81b-7562-4463-95ce-b35e8d65a664-kube-api-access-frxz6\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:08.619524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:08.619490    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:08.621907 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:08.621880    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:08.675946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:08.675914    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:09.009299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:09.009210    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"]
Oct 30 14:20:09.458314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:09.458274    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"]
Oct 30 14:20:09.470464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:09.470430    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s" event={"ID":"3a91d81b-7562-4463-95ce-b35e8d65a664","Type":"ContainerStarted","Data":"c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f"}
Oct 30 14:20:09.470651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:09.470488    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:09.470651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:09.470506    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s" event={"ID":"3a91d81b-7562-4463-95ce-b35e8d65a664","Type":"ContainerStarted","Data":"813e76033e86720ab6981dcefc658ce5db1d3fdadb405f09f7bf440d9ea77d69"}
Oct 30 14:20:09.491027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:09.491000    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"]
Oct 30 14:20:09.613582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:09.613428    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s" podStartSLOduration=2.287818206 podStartE2EDuration="2.613413748s" totalImagesPullingTime="325.595542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:09.610587524 +0000 UTC m=+1874.642486466" watchObservedRunningTime="2026-10-30 14:20:09.613413748 +0000 UTC m=+1874.645312739"
Oct 30 14:20:10.140601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.133228    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"]
Oct 30 14:20:10.142062 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.142033    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.142792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.142762    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"]
Oct 30 14:20:10.162809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.162787    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2a2835-serving-cert\""
Oct 30 14:20:10.194818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.194780    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-maas-api-tls\") pod \"maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.195029 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.194832    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2ksz\" (UniqueName: \"kubernetes.io/projected/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-kube-api-access-z2ksz\") pod \"maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.195112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.195024    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-metrics-certs\") pod \"maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.195179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.195122    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-openshift-service-ca\") pod \"maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.320564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.302515    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-openshift-service-ca\") pod \"maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.320564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.302747    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-maas-api-tls\") pod \"maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.320564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.302784    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ksz\" (UniqueName: \"kubernetes.io/projected/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-kube-api-access-z2ksz\") pod \"maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.320564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.302913    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-metrics-certs\") pod \"maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.320564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.309799    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-openshift-service-ca\") pod \"maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.336965 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.332192    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-metrics-certs\") pod \"maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.346412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.341223    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-maas-api-tls\") pod \"maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.365565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.361188    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ksz\" (UniqueName: \"kubernetes.io/projected/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-kube-api-access-z2ksz\") pod \"maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.472639 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.471835    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:10.906129 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:20:10.906101    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf224c0b8_2a3e_4dfc_b299_b9f03b5d118c.slice/crio-d01f41772abe66edd4e2775838951f75ef79fc8c302d157bec05fe0a31f920e4 WatchSource:0}: Error finding container d01f41772abe66edd4e2775838951f75ef79fc8c302d157bec05fe0a31f920e4: Status 404 returned error can't find the container with id d01f41772abe66edd4e2775838951f75ef79fc8c302d157bec05fe0a31f920e4
Oct 30 14:20:10.908345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:10.908324    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"]
Oct 30 14:20:11.375635 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:20:11.375596    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:20:11.552617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:11.550249    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr" event={"ID":"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c","Type":"ContainerStarted","Data":"f198113e982e906487d31c13b7d6d3a97ec65c43656cbe599e7bf04708b3414a"}
Oct 30 14:20:11.552617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:11.550319    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr" event={"ID":"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c","Type":"ContainerStarted","Data":"d01f41772abe66edd4e2775838951f75ef79fc8c302d157bec05fe0a31f920e4"}
Oct 30 14:20:11.552617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:11.551562    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:11.604720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:11.604664    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr" podStartSLOduration=2.231320858 podStartE2EDuration="2.604647493s" totalImagesPullingTime="373.326635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:11.601076599 +0000 UTC m=+1876.632975537" watchObservedRunningTime="2026-10-30 14:20:11.604647493 +0000 UTC m=+1876.636546430"
Oct 30 14:20:13.568771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:13.568739    4530 generic.go:356] "Generic (PLEG): container finished" podID="700073be-6e94-45fe-a2e3-d7505c506c85" containerID="72161d94197ddf52c4c8f10520677db3836631b7d791f0374d5430c1aa33aa84" exitCode=0
Oct 30 14:20:13.569298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:13.568771    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" event={"ID":"700073be-6e94-45fe-a2e3-d7505c506c85","Type":"ContainerDied","Data":"72161d94197ddf52c4c8f10520677db3836631b7d791f0374d5430c1aa33aa84"}
Oct 30 14:20:13.569395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:13.569296    4530 scope.go:122] "RemoveContainer" containerID="72161d94197ddf52c4c8f10520677db3836631b7d791f0374d5430c1aa33aa84"
Oct 30 14:20:13.570954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:13.570922    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:20:13.571055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:13.570992    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:20:13.571055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:13.571026    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:20:14.458758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.458702    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d" podUID="67fafa95-69f0-4942-a5e4-1843e3beb361" containerName="payload-processing" containerID="cri-o://fad847147c2a759d184f3b9d80e9b91fde6e62e20c80b73bc61864bf5db21976" gracePeriod=25
Oct 30 14:20:14.491367 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.491333    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw" podUID="44a34205-1dd8-44eb-89cb-e89a8f3c553e" containerName="payload-pre-processing" containerID="cri-o://c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664" gracePeriod=25
Oct 30 14:20:14.579060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.579031    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" event={"ID":"700073be-6e94-45fe-a2e3-d7505c506c85","Type":"ContainerStarted","Data":"bc785982c6bb96d790edeebaee3c6fd10d5df8a9e7b76e010b85b09b8062bbfa"}
Oct 30 14:20:14.580787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.580763    4530 generic.go:356] "Generic (PLEG): container finished" podID="67fafa95-69f0-4942-a5e4-1843e3beb361" containerID="fad847147c2a759d184f3b9d80e9b91fde6e62e20c80b73bc61864bf5db21976" exitCode=0
Oct 30 14:20:14.580898 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.580833    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d" event={"ID":"67fafa95-69f0-4942-a5e4-1843e3beb361","Type":"ContainerDied","Data":"fad847147c2a759d184f3b9d80e9b91fde6e62e20c80b73bc61864bf5db21976"}
Oct 30 14:20:14.870814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.870789    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"
Oct 30 14:20:14.874746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.874718    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"
Oct 30 14:20:14.940759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.939469    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67fafa95-69f0-4942-a5e4-1843e3beb361-kube-api-access-vhckv\" (UniqueName: \"kubernetes.io/projected/67fafa95-69f0-4942-a5e4-1843e3beb361-kube-api-access-vhckv\") pod \"67fafa95-69f0-4942-a5e4-1843e3beb361\" (UID: \"67fafa95-69f0-4942-a5e4-1843e3beb361\") "
Oct 30 14:20:14.940759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.939671    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a34205-1dd8-44eb-89cb-e89a8f3c553e-kube-api-access-gvb6f\" (UniqueName: \"kubernetes.io/projected/44a34205-1dd8-44eb-89cb-e89a8f3c553e-kube-api-access-gvb6f\") pod \"44a34205-1dd8-44eb-89cb-e89a8f3c553e\" (UID: \"44a34205-1dd8-44eb-89cb-e89a8f3c553e\") "
Oct 30 14:20:14.940759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.939769    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67fafa95-69f0-4942-a5e4-1843e3beb361-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fafa95-69f0-4942-a5e4-1843e3beb361-plugins-config-volume\") pod \"67fafa95-69f0-4942-a5e4-1843e3beb361\" (UID: \"67fafa95-69f0-4942-a5e4-1843e3beb361\") "
Oct 30 14:20:14.940759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.939837    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a34205-1dd8-44eb-89cb-e89a8f3c553e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a34205-1dd8-44eb-89cb-e89a8f3c553e-plugins-config-volume\") pod \"44a34205-1dd8-44eb-89cb-e89a8f3c553e\" (UID: \"44a34205-1dd8-44eb-89cb-e89a8f3c553e\") "
Oct 30 14:20:14.940759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.940315    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fafa95-69f0-4942-a5e4-1843e3beb361-plugins-config-volume" pod "67fafa95-69f0-4942-a5e4-1843e3beb361" (UID: "67fafa95-69f0-4942-a5e4-1843e3beb361"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:20:14.940759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.940318    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a34205-1dd8-44eb-89cb-e89a8f3c553e-plugins-config-volume" pod "44a34205-1dd8-44eb-89cb-e89a8f3c553e" (UID: "44a34205-1dd8-44eb-89cb-e89a8f3c553e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:20:14.940759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.940684    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fafa95-69f0-4942-a5e4-1843e3beb361-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:14.940759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.940705    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a34205-1dd8-44eb-89cb-e89a8f3c553e-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:14.942195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.942169    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fafa95-69f0-4942-a5e4-1843e3beb361-kube-api-access-vhckv" pod "67fafa95-69f0-4942-a5e4-1843e3beb361" (UID: "67fafa95-69f0-4942-a5e4-1843e3beb361"). InnerVolumeSpecName "kube-api-access-vhckv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:20:14.943839 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:14.943808    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a34205-1dd8-44eb-89cb-e89a8f3c553e-kube-api-access-gvb6f" pod "44a34205-1dd8-44eb-89cb-e89a8f3c553e" (UID: "44a34205-1dd8-44eb-89cb-e89a8f3c553e"). InnerVolumeSpecName "kube-api-access-gvb6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:20:15.041892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.041811    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhckv\" (UniqueName: \"kubernetes.io/projected/67fafa95-69f0-4942-a5e4-1843e3beb361-kube-api-access-vhckv\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:15.041892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.041840    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvb6f\" (UniqueName: \"kubernetes.io/projected/44a34205-1dd8-44eb-89cb-e89a8f3c553e-kube-api-access-gvb6f\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:15.337865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.337832    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:20:15.339376 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.339346    4530 patch_prober.go:29] interesting pod/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 14:20:15.339506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.339405    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 14:20:15.511172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.511126    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:20:15.516992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.516964    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:20:15.616676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.616533    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d" event={"ID":"67fafa95-69f0-4942-a5e4-1843e3beb361","Type":"ContainerDied","Data":"3d545a250b06a07af5b475784309aab3a783f887697f9c6a79728de13f1dddc6"}
Oct 30 14:20:15.616676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.616652    4530 scope.go:122] "RemoveContainer" containerID="fad847147c2a759d184f3b9d80e9b91fde6e62e20c80b73bc61864bf5db21976"
Oct 30 14:20:15.617375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.617352    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"
Oct 30 14:20:15.623425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.621498    4530 generic.go:356] "Generic (PLEG): container finished" podID="44a34205-1dd8-44eb-89cb-e89a8f3c553e" containerID="c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664" exitCode=0
Oct 30 14:20:15.623425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.622684    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"
Oct 30 14:20:15.623425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.623208    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw" event={"ID":"44a34205-1dd8-44eb-89cb-e89a8f3c553e","Type":"ContainerDied","Data":"c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664"}
Oct 30 14:20:15.623425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.623250    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw" event={"ID":"44a34205-1dd8-44eb-89cb-e89a8f3c553e","Type":"ContainerDied","Data":"5cbf8da0186bfbd5bc24cc0d9247cec0a7efd368394f41045698034bc91bf2e5"}
Oct 30 14:20:15.633517 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.633500    4530 scope.go:122] "RemoveContainer" containerID="c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664"
Oct 30 14:20:15.645710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.645690    4530 scope.go:122] "RemoveContainer" containerID="c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664"
Oct 30 14:20:15.646065 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:20:15.646039    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664\": container with ID starting with c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664 not found: ID does not exist" containerID="c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664"
Oct 30 14:20:15.646126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.646076    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664"} err="failed to get container status \"c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664\": rpc error: code = NotFound desc = could not find container \"c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664\": container with ID starting with c41a459423b240f6a37bbed71766332979c7d2452d0d82387521e57a1cf89664 not found: ID does not exist"
Oct 30 14:20:15.713637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.713603    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"]
Oct 30 14:20:15.718417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.718383    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-x5f5d"]
Oct 30 14:20:15.850107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.850041    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"]
Oct 30 14:20:15.907799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:15.898354    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-h5tsw"]
Oct 30 14:20:16.352563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:16.338293    4530 patch_prober.go:29] interesting pod/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 14:20:16.352563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:16.338374    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 14:20:17.117393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:17.117350    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a34205-1dd8-44eb-89cb-e89a8f3c553e" path="/var/lib/kubelet/pods/44a34205-1dd8-44eb-89cb-e89a8f3c553e/volumes"
Oct 30 14:20:17.117957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:17.117941    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67fafa95-69f0-4942-a5e4-1843e3beb361" path="/var/lib/kubelet/pods/67fafa95-69f0-4942-a5e4-1843e3beb361/volumes"
Oct 30 14:20:17.338417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:17.338378    4530 patch_prober.go:29] interesting pod/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 14:20:17.338626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:17.338436    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 14:20:18.338226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:18.338190    4530 patch_prober.go:29] interesting pod/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 14:20:18.338645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:18.338269    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 14:20:18.574158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:18.574127    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:20:19.338497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:19.338460    4530 patch_prober.go:29] interesting pod/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 14:20:19.338946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:19.338519    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 14:20:20.338761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:20.338720    4530 patch_prober.go:29] interesting pod/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 14:20:20.339162 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:20.338786    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 14:20:21.343592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:21.343561    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:20:21.344605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:21.344580    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:20:23.227211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:23.227171    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:20:25.658185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:25.658152    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"]
Oct 30 14:20:25.658665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:25.658364    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw" podUID="6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b" containerName="maas-api" containerID="cri-o://7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a" gracePeriod=30
Oct 30 14:20:26.161233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.160355    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:20:26.377779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.368136    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-maas-api-tls\") pod \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") "
Oct 30 14:20:26.377779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.368205    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-metrics-certs\") pod \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") "
Oct 30 14:20:26.377779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.368279    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-kube-api-access-5rxx7\" (UniqueName: \"kubernetes.io/projected/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-kube-api-access-5rxx7\") pod \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") "
Oct 30 14:20:26.377779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.368329    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-openshift-service-ca\") pod \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\" (UID: \"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b\") "
Oct 30 14:20:26.377779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.372771    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-openshift-service-ca" pod "6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b" (UID: "6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:20:26.411562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.384666    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-maas-api-tls" pod "6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b" (UID: "6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:20:26.481564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.475076    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:26.481564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.475119    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:26.564568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.552730    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-metrics-certs" pod "6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b" (UID: "6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:20:26.581565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.578118    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:26.615959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.613779    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-kube-api-access-5rxx7" pod "6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b" (UID: "6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b"). InnerVolumeSpecName "kube-api-access-5rxx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:20:26.703714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.702137    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rxx7\" (UniqueName: \"kubernetes.io/projected/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b-kube-api-access-5rxx7\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:26.837460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.837278    4530 generic.go:356] "Generic (PLEG): container finished" podID="6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b" containerID="7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a" exitCode=0
Oct 30 14:20:26.837760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.837488    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw" event={"ID":"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b","Type":"ContainerDied","Data":"7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a"}
Oct 30 14:20:26.837760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.837531    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw" event={"ID":"6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b","Type":"ContainerDied","Data":"bef0c3cf32403fde1088d5ebde187edc124f34450ae5fce4ee1529b72e9763e7"}
Oct 30 14:20:26.837760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.837585    4530 scope.go:122] "RemoveContainer" containerID="7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a"
Oct 30 14:20:26.837760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.837756    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"
Oct 30 14:20:26.852189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.852170    4530 scope.go:122] "RemoveContainer" containerID="7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a"
Oct 30 14:20:26.852453 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:20:26.852435    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a\": container with ID starting with 7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a not found: ID does not exist" containerID="7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a"
Oct 30 14:20:26.852516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.852461    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a"} err="failed to get container status \"7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a\": rpc error: code = NotFound desc = could not find container \"7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a\": container with ID starting with 7bb86db09c7814fd4309e9a384743e39c991e8e195b8c94b5c98c16c68aae52a not found: ID does not exist"
Oct 30 14:20:26.910198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.910167    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"]
Oct 30 14:20:26.918662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:26.918635    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9dba4af9-7f67499d99-fp8jw"]
Oct 30 14:20:27.119174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:27.119141    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b" path="/var/lib/kubelet/pods/6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b/volumes"
Oct 30 14:20:34.367889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.367854    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"]
Oct 30 14:20:34.368624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.368606    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b" containerName="maas-api"
Oct 30 14:20:34.368678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.368625    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b" containerName="maas-api"
Oct 30 14:20:34.368678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.368651    4530 cpu_manager.go:414] "containerMap: removing container" podUID="67fafa95-69f0-4942-a5e4-1843e3beb361" containerName="payload-processing"
Oct 30 14:20:34.368678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.368656    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67fafa95-69f0-4942-a5e4-1843e3beb361" containerName="payload-processing"
Oct 30 14:20:34.368678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.368669    4530 cpu_manager.go:414] "containerMap: removing container" podUID="44a34205-1dd8-44eb-89cb-e89a8f3c553e" containerName="payload-pre-processing"
Oct 30 14:20:34.368678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.368675    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a34205-1dd8-44eb-89cb-e89a8f3c553e" containerName="payload-pre-processing"
Oct 30 14:20:34.368838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.368797    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="67fafa95-69f0-4942-a5e4-1843e3beb361" containerName="payload-processing"
Oct 30 14:20:34.368838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.368809    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a34205-1dd8-44eb-89cb-e89a8f3c553e" containerName="payload-pre-processing"
Oct 30 14:20:34.368838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.368821    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e15d898-0f9b-4f03-8bc0-e33f8c6fce5b" containerName="maas-api"
Oct 30 14:20:34.372953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.372934    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.393646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.393622    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5a9d7b-serving-cert\""
Oct 30 14:20:34.396273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.396248    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"]
Oct 30 14:20:34.477418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.477385    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-metrics-certs\") pod \"maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.477611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.477447    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/736851b6-cadd-4b44-964c-57cd97372c82-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.477797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.477773    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-maas-api-tls\") pod \"maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.477872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.477839    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdzk6\" (UniqueName: \"kubernetes.io/projected/736851b6-cadd-4b44-964c-57cd97372c82-kube-api-access-cdzk6\") pod \"maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.594512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.594472    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-maas-api-tls\") pod \"maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.594885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.594856    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdzk6\" (UniqueName: \"kubernetes.io/projected/736851b6-cadd-4b44-964c-57cd97372c82-kube-api-access-cdzk6\") pod \"maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.595785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.595743    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-metrics-certs\") pod \"maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.595910 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.595821    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/736851b6-cadd-4b44-964c-57cd97372c82-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.596730 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.596705    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/736851b6-cadd-4b44-964c-57cd97372c82-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.601622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.601595    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-maas-api-tls\") pod \"maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.618931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.618689    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-metrics-certs\") pod \"maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.626189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.626156    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdzk6\" (UniqueName: \"kubernetes.io/projected/736851b6-cadd-4b44-964c-57cd97372c82-kube-api-access-cdzk6\") pod \"maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:34.709802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:34.709487    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:35.037951 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:20:35.037923    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736851b6_cadd_4b44_964c_57cd97372c82.slice/crio-237f8296b901cdaa7cb52ab0bcec2d703652fef22d34c82f617f5de69f9fac58 WatchSource:0}: Error finding container 237f8296b901cdaa7cb52ab0bcec2d703652fef22d34c82f617f5de69f9fac58: Status 404 returned error can't find the container with id 237f8296b901cdaa7cb52ab0bcec2d703652fef22d34c82f617f5de69f9fac58
Oct 30 14:20:35.040629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:35.040599    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"]
Oct 30 14:20:35.931304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:35.931252    4530 generic.go:356] "Generic (PLEG): container finished" podID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerID="9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22" exitCode=0
Oct 30 14:20:35.931866 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:35.931335    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" event={"ID":"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a","Type":"ContainerDied","Data":"9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22"}
Oct 30 14:20:35.932955 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:35.932152    4530 scope.go:122] "RemoveContainer" containerID="9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22"
Oct 30 14:20:35.934664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:35.934574    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97" event={"ID":"736851b6-cadd-4b44-964c-57cd97372c82","Type":"ContainerStarted","Data":"76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38"}
Oct 30 14:20:35.934664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:35.934632    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97" event={"ID":"736851b6-cadd-4b44-964c-57cd97372c82","Type":"ContainerStarted","Data":"237f8296b901cdaa7cb52ab0bcec2d703652fef22d34c82f617f5de69f9fac58"}
Oct 30 14:20:35.936768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:35.935032    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:20:35.936768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:35.935113    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:20:35.936768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:35.935150    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:20:35.937926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:35.937908    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:36.056290 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.056230    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97" podStartSLOduration=1.718673047 podStartE2EDuration="2.056210742s" totalImagesPullingTime="337.537695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:36.052059246 +0000 UTC m=+1901.083958199" watchObservedRunningTime="2026-10-30 14:20:36.056210742 +0000 UTC m=+1901.088109681"
Oct 30 14:20:36.314007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.313972    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"]
Oct 30 14:20:36.317859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.317833    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.338183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.338141    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4bf33a-serving-cert\""
Oct 30 14:20:36.346732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.344489    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f414f83-9288-4b7d-9e7b-1170aa2beab0-openshift-service-ca\") pod \"maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.346732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.345678    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-metrics-certs\") pod \"maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.346732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.345928    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwrrq\" (UniqueName: \"kubernetes.io/projected/4f414f83-9288-4b7d-9e7b-1170aa2beab0-kube-api-access-vwrrq\") pod \"maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.346732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.346035    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-maas-api-tls\") pod \"maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.347858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.347827    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"]
Oct 30 14:20:36.421701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.421673    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:20:36.447286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.447251    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-maas-api-tls\") pod \"maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.447458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.447406    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f414f83-9288-4b7d-9e7b-1170aa2beab0-openshift-service-ca\") pod \"maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.447556 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.447517    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-metrics-certs\") pod \"maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.447748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.447720    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwrrq\" (UniqueName: \"kubernetes.io/projected/4f414f83-9288-4b7d-9e7b-1170aa2beab0-kube-api-access-vwrrq\") pod \"maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.448310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.448285    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f414f83-9288-4b7d-9e7b-1170aa2beab0-openshift-service-ca\") pod \"maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.450157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.450127    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-metrics-certs\") pod \"maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.450307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.450289    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-maas-api-tls\") pod \"maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.476448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.476425    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwrrq\" (UniqueName: \"kubernetes.io/projected/4f414f83-9288-4b7d-9e7b-1170aa2beab0-kube-api-access-vwrrq\") pod \"maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.632970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.632861    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:36.945621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:36.945484    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" event={"ID":"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a","Type":"ContainerStarted","Data":"445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26"}
Oct 30 14:20:37.010898 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:20:37.010864    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f414f83_9288_4b7d_9e7b_1170aa2beab0.slice/crio-83945db42ecb3b6164edbe20208a7bb1a895bcf912950e8684d0af5be5bbffab WatchSource:0}: Error finding container 83945db42ecb3b6164edbe20208a7bb1a895bcf912950e8684d0af5be5bbffab: Status 404 returned error can't find the container with id 83945db42ecb3b6164edbe20208a7bb1a895bcf912950e8684d0af5be5bbffab
Oct 30 14:20:37.011534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:37.011511    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"]
Oct 30 14:20:37.609296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:37.609176    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:20:37.610669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:37.610639    4530 patch_prober.go:29] interesting pod/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 14:20:37.610820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:37.610698    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 14:20:37.957079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:37.956992    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29" event={"ID":"4f414f83-9288-4b7d-9e7b-1170aa2beab0","Type":"ContainerStarted","Data":"93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1"}
Oct 30 14:20:37.957079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:37.957035    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29" event={"ID":"4f414f83-9288-4b7d-9e7b-1170aa2beab0","Type":"ContainerStarted","Data":"83945db42ecb3b6164edbe20208a7bb1a895bcf912950e8684d0af5be5bbffab"}
Oct 30 14:20:37.957079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:37.957078    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:38.022563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:38.013102    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29" podStartSLOduration=1.4719921999999999 podStartE2EDuration="2.013081872s" totalImagesPullingTime="541.089672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:38.009961124 +0000 UTC m=+1903.041860059" watchObservedRunningTime="2026-10-30 14:20:38.013081872 +0000 UTC m=+1903.044980816"
Oct 30 14:20:38.622320 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:38.609905    4530 patch_prober.go:29] interesting pod/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 14:20:38.622320 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:38.609968    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 14:20:39.608813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:39.608777    4530 patch_prober.go:29] interesting pod/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 14:20:39.609299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:39.608846    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 14:20:40.618564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:40.612093    4530 patch_prober.go:29] interesting pod/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 14:20:40.618564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:40.612159    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 14:20:41.368600 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:20:41.368566    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:20:41.608614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:41.608507    4530 patch_prober.go:29] interesting pod/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 14:20:41.608614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:41.608578    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 14:20:41.952334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:41.952302    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:20:42.055831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:42.055792    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"]
Oct 30 14:20:42.613111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:42.613081    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:20:42.613602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:42.613571    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:20:42.614272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:42.614252    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:20:43.974419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:43.974393    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:20:56.914833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:56.914789    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"]
Oct 30 14:20:56.915309 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:56.915000    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" podUID="259f2601-7ea2-43b0-abeb-73891558a746" containerName="istio-proxy" containerID="cri-o://2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6" gracePeriod=30
Oct 30 14:20:57.126940 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.126875    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr" podUID="91d82203-dd8e-4e1f-ad76-0403aea6eaca" containerName="main" containerID="cri-o://3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4" gracePeriod=45
Oct 30 14:20:57.519415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.519350    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:20:57.564005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.563974    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-home\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-home\") pod \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") "
Oct 30 14:20:57.564176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.564030    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tmp-dir\") pod \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") "
Oct 30 14:20:57.564176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.564170    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tls-certs\" (UniqueName: \"kubernetes.io/secret/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tls-certs\") pod \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") "
Oct 30 14:20:57.564303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.564252    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-dshm\") pod \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") "
Oct 30 14:20:57.564382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.564359    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tmp-dir" pod "91d82203-dd8e-4e1f-ad76-0403aea6eaca" (UID: "91d82203-dd8e-4e1f-ad76-0403aea6eaca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:20:57.564466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.564447    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d82203-dd8e-4e1f-ad76-0403aea6eaca-kube-api-access-zr7w7\" (UniqueName: \"kubernetes.io/projected/91d82203-dd8e-4e1f-ad76-0403aea6eaca-kube-api-access-zr7w7\") pod \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") "
Oct 30 14:20:57.564512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.564502    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-model-cache\") pod \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\" (UID: \"91d82203-dd8e-4e1f-ad76-0403aea6eaca\") "
Oct 30 14:20:57.564722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.564447    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-home" pod "91d82203-dd8e-4e1f-ad76-0403aea6eaca" (UID: "91d82203-dd8e-4e1f-ad76-0403aea6eaca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:20:57.565036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.565000    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-model-cache" pod "91d82203-dd8e-4e1f-ad76-0403aea6eaca" (UID: "91d82203-dd8e-4e1f-ad76-0403aea6eaca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:20:57.565273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.565256    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:57.565365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.565274    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:57.565365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.565284    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:57.567360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.567318    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d82203-dd8e-4e1f-ad76-0403aea6eaca-kube-api-access-zr7w7" pod "91d82203-dd8e-4e1f-ad76-0403aea6eaca" (UID: "91d82203-dd8e-4e1f-ad76-0403aea6eaca"). InnerVolumeSpecName "kube-api-access-zr7w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:20:57.567679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.567647    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tls-certs" pod "91d82203-dd8e-4e1f-ad76-0403aea6eaca" (UID: "91d82203-dd8e-4e1f-ad76-0403aea6eaca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:20:57.572000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.571971    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-dshm" pod "91d82203-dd8e-4e1f-ad76-0403aea6eaca" (UID: "91d82203-dd8e-4e1f-ad76-0403aea6eaca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:20:57.666720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.666688    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr7w7\" (UniqueName: \"kubernetes.io/projected/91d82203-dd8e-4e1f-ad76-0403aea6eaca-kube-api-access-zr7w7\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:57.666720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.666716    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d82203-dd8e-4e1f-ad76-0403aea6eaca-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:57.666720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:57.666728    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d82203-dd8e-4e1f-ad76-0403aea6eaca-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:20:58.509347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:58.509295    4530 generic.go:356] "Generic (PLEG): container finished" podID="91d82203-dd8e-4e1f-ad76-0403aea6eaca" containerID="3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4" exitCode=0
Oct 30 14:20:58.510302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:58.509777    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr" event={"ID":"91d82203-dd8e-4e1f-ad76-0403aea6eaca","Type":"ContainerDied","Data":"3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4"}
Oct 30 14:20:58.510422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:58.510334    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr" event={"ID":"91d82203-dd8e-4e1f-ad76-0403aea6eaca","Type":"ContainerDied","Data":"36411decd111e87cbda651610384474b606011e932e572bbd713cfa7bd847d64"}
Oct 30 14:20:58.510422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:58.510381    4530 scope.go:122] "RemoveContainer" containerID="3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4"
Oct 30 14:20:58.510776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:58.509904    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"
Oct 30 14:20:58.523772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:58.523754    4530 scope.go:122] "RemoveContainer" containerID="3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4"
Oct 30 14:20:58.524102 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:20:58.524072    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4\": container with ID starting with 3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4 not found: ID does not exist" containerID="3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4"
Oct 30 14:20:58.524178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:58.524107    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4"} err="failed to get container status \"3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4\": rpc error: code = NotFound desc = could not find container \"3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4\": container with ID starting with 3521cf3b133841c106ded54e753193f0522ae2083d0d0fa2ee3d030c3540c7c4 not found: ID does not exist"
Oct 30 14:20:58.575588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:58.575559    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"]
Oct 30 14:20:58.605600 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:58.605555    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9dba4af9/e2e-lifecycle-model-9dba4af9-kserve-74b85c4bcc-6mzgr"]
Oct 30 14:20:59.188524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:20:59.188485    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d82203-dd8e-4e1f-ad76-0403aea6eaca" path="/var/lib/kubelet/pods/91d82203-dd8e-4e1f-ad76-0403aea6eaca/volumes"
Oct 30 14:21:01.359274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:01.359106    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:21:01.359274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:01.359207    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" podUID="259f2601-7ea2-43b0-abeb-73891558a746" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:21:02.544921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.544631    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:21:02.649232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.648736    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-socket\") pod \"259f2601-7ea2-43b0-abeb-73891558a746\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") "
Oct 30 14:21:02.649232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.648846    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-certs\") pod \"259f2601-7ea2-43b0-abeb-73891558a746\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") "
Oct 30 14:21:02.649232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.648946    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-envoy\") pod \"259f2601-7ea2-43b0-abeb-73891558a746\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") "
Oct 30 14:21:02.649232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.649023    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-data\") pod \"259f2601-7ea2-43b0-abeb-73891558a746\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") "
Oct 30 14:21:02.649232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.649058    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-credential-socket\") pod \"259f2601-7ea2-43b0-abeb-73891558a746\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") "
Oct 30 14:21:02.649232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.649123    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-kube-api-access-q6hnk\" (UniqueName: \"kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-kube-api-access-q6hnk\") pod \"259f2601-7ea2-43b0-abeb-73891558a746\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") "
Oct 30 14:21:02.649742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.649281    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/259f2601-7ea2-43b0-abeb-73891558a746-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259f2601-7ea2-43b0-abeb-73891558a746-istio-podinfo\") pod \"259f2601-7ea2-43b0-abeb-73891558a746\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") "
Oct 30 14:21:02.649742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.649316    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-istio-token\" (UniqueName: \"kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-istio-token\") pod \"259f2601-7ea2-43b0-abeb-73891558a746\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") "
Oct 30 14:21:02.649742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.649345    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259f2601-7ea2-43b0-abeb-73891558a746-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259f2601-7ea2-43b0-abeb-73891558a746-istiod-ca-cert\") pod \"259f2601-7ea2-43b0-abeb-73891558a746\" (UID: \"259f2601-7ea2-43b0-abeb-73891558a746\") "
Oct 30 14:21:02.652683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.652634    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-data" pod "259f2601-7ea2-43b0-abeb-73891558a746" (UID: "259f2601-7ea2-43b0-abeb-73891558a746"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:02.653421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.653378    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-credential-socket" pod "259f2601-7ea2-43b0-abeb-73891558a746" (UID: "259f2601-7ea2-43b0-abeb-73891558a746"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:02.653637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.653605    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-certs" pod "259f2601-7ea2-43b0-abeb-73891558a746" (UID: "259f2601-7ea2-43b0-abeb-73891558a746"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:02.653782 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.653756    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-socket" pod "259f2601-7ea2-43b0-abeb-73891558a746" (UID: "259f2601-7ea2-43b0-abeb-73891558a746"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:02.658124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.658084    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259f2601-7ea2-43b0-abeb-73891558a746-istiod-ca-cert" pod "259f2601-7ea2-43b0-abeb-73891558a746" (UID: "259f2601-7ea2-43b0-abeb-73891558a746"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:21:02.658333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.658297    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-kube-api-access-q6hnk" pod "259f2601-7ea2-43b0-abeb-73891558a746" (UID: "259f2601-7ea2-43b0-abeb-73891558a746"). InnerVolumeSpecName "kube-api-access-q6hnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:02.659152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.659071    4530 generic.go:356] "Generic (PLEG): container finished" podID="259f2601-7ea2-43b0-abeb-73891558a746" containerID="2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6" exitCode=0
Oct 30 14:21:02.659278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.659218    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/259f2601-7ea2-43b0-abeb-73891558a746-istio-podinfo" pod "259f2601-7ea2-43b0-abeb-73891558a746" (UID: "259f2601-7ea2-43b0-abeb-73891558a746"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:21:02.659389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.659309    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"
Oct 30 14:21:02.659389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.659371    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" event={"ID":"259f2601-7ea2-43b0-abeb-73891558a746","Type":"ContainerDied","Data":"2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6"}
Oct 30 14:21:02.659528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.659422    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6" event={"ID":"259f2601-7ea2-43b0-abeb-73891558a746","Type":"ContainerDied","Data":"553f11896397f6284a4ddbe31cc729070c1a6c85343f36de4c2f39220ca1c823"}
Oct 30 14:21:02.659528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.659456    4530 scope.go:122] "RemoveContainer" containerID="2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6"
Oct 30 14:21:02.662437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.661597    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-istio-token" pod "259f2601-7ea2-43b0-abeb-73891558a746" (UID: "259f2601-7ea2-43b0-abeb-73891558a746"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:02.665634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.665606    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-envoy" pod "259f2601-7ea2-43b0-abeb-73891558a746" (UID: "259f2601-7ea2-43b0-abeb-73891558a746"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:02.746771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.746742    4530 scope.go:122] "RemoveContainer" containerID="2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6"
Oct 30 14:21:02.747219 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:21:02.747188    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6\": container with ID starting with 2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6 not found: ID does not exist" containerID="2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6"
Oct 30 14:21:02.747310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.747230    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6"} err="failed to get container status \"2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6\": rpc error: code = NotFound desc = could not find container \"2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6\": container with ID starting with 2a4d4c2e0fa62bda26d4eef14ac96953d090c4f0a5216f2ae3246853ada009c6 not found: ID does not exist"
Oct 30 14:21:02.754310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.754275    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:02.754310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.754310    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hnk\" (UniqueName: \"kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-kube-api-access-q6hnk\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:02.754494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.754327    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259f2601-7ea2-43b0-abeb-73891558a746-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:02.754494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.754341    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259f2601-7ea2-43b0-abeb-73891558a746-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:02.754494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.754356    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259f2601-7ea2-43b0-abeb-73891558a746-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:02.754494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.754368    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:02.754494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.754380    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:02.754494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.754393    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:02.754494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:02.754404    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259f2601-7ea2-43b0-abeb-73891558a746-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:03.073468 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:03.073432    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"]
Oct 30 14:21:03.130078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:03.130012    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-dgbv6"]
Oct 30 14:21:03.165708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:03.165677    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="259f2601-7ea2-43b0-abeb-73891558a746" path="/var/lib/kubelet/pods/259f2601-7ea2-43b0-abeb-73891558a746/volumes"
Oct 30 14:21:07.179742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:07.179692    4530 patch_prober.go:29] interesting pod/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:21:07.180251 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:07.179800    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:21:10.777180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:10.777088    4530 generic.go:356] "Generic (PLEG): container finished" podID="72a45514-45c1-441e-ad52-b48b6072eac8" containerID="b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94" exitCode=0
Oct 30 14:21:10.777706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:10.777173    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" event={"ID":"72a45514-45c1-441e-ad52-b48b6072eac8","Type":"ContainerDied","Data":"b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94"}
Oct 30 14:21:10.777781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:10.777740    4530 scope.go:122] "RemoveContainer" containerID="b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94"
Oct 30 14:21:10.779649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:10.779621    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:10.779732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:10.779691    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:10.779770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:10.779743    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:11.376036 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:21:11.375978    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:21:11.788976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.788941    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" event={"ID":"72a45514-45c1-441e-ad52-b48b6072eac8","Type":"ContainerStarted","Data":"76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a"}
Oct 30 14:21:11.790799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.790773    4530 generic.go:356] "Generic (PLEG): container finished" podID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerID="7a86d1bebd2b06b31b251cda9d9c22c43c4533959a6f0a34b5a6df2a866bbb43" exitCode=0
Oct 30 14:21:11.790935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.790856    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" event={"ID":"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472","Type":"ContainerDied","Data":"7a86d1bebd2b06b31b251cda9d9c22c43c4533959a6f0a34b5a6df2a866bbb43"}
Oct 30 14:21:11.791278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.791260    4530 scope.go:122] "RemoveContainer" containerID="7a86d1bebd2b06b31b251cda9d9c22c43c4533959a6f0a34b5a6df2a866bbb43"
Oct 30 14:21:11.792871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.792849    4530 generic.go:356] "Generic (PLEG): container finished" podID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerID="029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9" exitCode=0
Oct 30 14:21:11.793010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.792922    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" event={"ID":"cdc89bfd-e9d1-4821-883c-18f959c7793f","Type":"ContainerDied","Data":"029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9"}
Oct 30 14:21:11.793353 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.793336    4530 scope.go:122] "RemoveContainer" containerID="029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9"
Oct 30 14:21:11.793419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.793345    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:11.793479 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.793429    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:11.793536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.793473    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:11.795225 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.795195    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:11.795336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.795273    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:11.795336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:11.795323    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:12.214289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:12.214245    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:21:12.282791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:12.282748    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:21:12.454571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:12.444482    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 14:21:12.454571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:12.444572    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 14:21:12.819930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:12.819893    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" event={"ID":"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472","Type":"ContainerStarted","Data":"0422235b73ff130d48d4f305cf874f7ecacceb8e163252898771972b34be32aa"}
Oct 30 14:21:12.822704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:12.822672    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" event={"ID":"cdc89bfd-e9d1-4821-883c-18f959c7793f","Type":"ContainerStarted","Data":"d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3"}
Oct 30 14:21:12.912393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:12.911844    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:21:12.912393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:12.912093    4530 patch_prober.go:29] interesting pod/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 14:21:12.912393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:12.912148    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 14:21:13.151625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:13.150970    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 14:21:13.151625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:13.151038    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 14:21:13.641566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:13.641052    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:21:13.641566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:13.641194    4530 patch_prober.go:29] interesting pod/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 14:21:13.641566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:13.641242    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 14:21:13.947645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:13.939468    4530 patch_prober.go:29] interesting pod/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 14:21:13.947645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:13.939528    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 14:21:14.157874 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:14.157828    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 14:21:14.158086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:14.157909    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 14:21:14.644565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:14.643694    4530 patch_prober.go:29] interesting pod/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 14:21:14.644565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:14.643755    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 14:21:14.914039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:14.913937    4530 patch_prober.go:29] interesting pod/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 14:21:14.914292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:14.914262    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 14:21:15.165331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:15.165037    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 14:21:15.165331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:15.165103    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 14:21:15.642564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:15.642021    4530 patch_prober.go:29] interesting pod/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 14:21:15.642564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:15.642087    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 14:21:15.916710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:15.913642    4530 patch_prober.go:29] interesting pod/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 14:21:15.916710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:15.913705    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 14:21:16.166593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:16.162840    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 14:21:16.166593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:16.162896    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 14:21:16.658061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:16.658012    4530 patch_prober.go:29] interesting pod/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 14:21:16.658254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:16.658077    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 14:21:16.965288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:16.965195    4530 patch_prober.go:29] interesting pod/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 14:21:16.966866 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:16.966819    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 14:21:17.164647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:17.155317    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 14:21:17.164647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:17.162782    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 14:21:17.787565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:17.658452    4530 patch_prober.go:29] interesting pod/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 14:21:17.787565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:17.658531    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 14:21:17.954783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:17.954742    4530 patch_prober.go:29] interesting pod/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 14:21:17.955086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:17.955054    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 14:21:18.150514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:18.150475    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 14:21:18.165184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:18.164629    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 14:21:18.643116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:18.643080    4530 patch_prober.go:29] interesting pod/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 14:21:18.643416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:18.643386    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 14:21:18.915077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:18.912954    4530 patch_prober.go:29] interesting pod/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 14:21:18.915077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:18.913021    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 14:21:19.074356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:19.074304    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:21:19.155153 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:19.155117    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:21:19.156153 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:19.156126    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:21:19.616997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:19.616958    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"]
Oct 30 14:21:19.666571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:19.639967    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw" podUID="3203b2df-5ed6-41af-b1a5-341fdbfd92a5" containerName="maas-api" containerID="cri-o://9059ad836f80bec2d84afb3006ea0c1dc4b8de01614d69743594fd22b4e5850f" gracePeriod=30
Oct 30 14:21:20.024668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.024635    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:21:20.031391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.031325    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:21:20.313659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.313620    4530 generic.go:356] "Generic (PLEG): container finished" podID="3203b2df-5ed6-41af-b1a5-341fdbfd92a5" containerID="9059ad836f80bec2d84afb3006ea0c1dc4b8de01614d69743594fd22b4e5850f" exitCode=0
Oct 30 14:21:20.315097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.315063    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw" event={"ID":"3203b2df-5ed6-41af-b1a5-341fdbfd92a5","Type":"ContainerDied","Data":"9059ad836f80bec2d84afb3006ea0c1dc4b8de01614d69743594fd22b4e5850f"}
Oct 30 14:21:20.511131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.511095    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:21:20.512819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.512732    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:21:20.517196 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.517174    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:21:20.521586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.521430    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:21:20.595296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.595256    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-maas-api-tls\") pod \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") "
Oct 30 14:21:20.601867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.601818    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-maas-api-tls" pod "3203b2df-5ed6-41af-b1a5-341fdbfd92a5" (UID: "3203b2df-5ed6-41af-b1a5-341fdbfd92a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:21:20.697212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.697180    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-metrics-certs\") pod \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") "
Oct 30 14:21:20.697398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.697238    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-openshift-service-ca\") pod \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") "
Oct 30 14:21:20.697471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.697443    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-kube-api-access-gpgz8\" (UniqueName: \"kubernetes.io/projected/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-kube-api-access-gpgz8\") pod \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\" (UID: \"3203b2df-5ed6-41af-b1a5-341fdbfd92a5\") "
Oct 30 14:21:20.697691 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.697663    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-openshift-service-ca" pod "3203b2df-5ed6-41af-b1a5-341fdbfd92a5" (UID: "3203b2df-5ed6-41af-b1a5-341fdbfd92a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:21:20.698106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.698078    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:20.698106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.698105    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:20.699815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.699782    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-metrics-certs" pod "3203b2df-5ed6-41af-b1a5-341fdbfd92a5" (UID: "3203b2df-5ed6-41af-b1a5-341fdbfd92a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:21:20.701199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.701176    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-kube-api-access-gpgz8" pod "3203b2df-5ed6-41af-b1a5-341fdbfd92a5" (UID: "3203b2df-5ed6-41af-b1a5-341fdbfd92a5"). InnerVolumeSpecName "kube-api-access-gpgz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:20.801286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.801201    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:20.801286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:20.801242    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpgz8\" (UniqueName: \"kubernetes.io/projected/3203b2df-5ed6-41af-b1a5-341fdbfd92a5-kube-api-access-gpgz8\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:21.323628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:21.323594    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"
Oct 30 14:21:21.323628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:21.323605    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw" event={"ID":"3203b2df-5ed6-41af-b1a5-341fdbfd92a5","Type":"ContainerDied","Data":"3bb4917acb1ba8659444db58625aa4ee15ca15802cf0d506cf62877dd48786f4"}
Oct 30 14:21:21.324180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:21.323664    4530 scope.go:122] "RemoveContainer" containerID="9059ad836f80bec2d84afb3006ea0c1dc4b8de01614d69743594fd22b4e5850f"
Oct 30 14:21:21.386876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:21.386798    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"]
Oct 30 14:21:21.400802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:21.400765    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d727dc-64f9f46b9f-ztzqw"]
Oct 30 14:21:21.529623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:21.529577    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:21:23.120533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:23.120492    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3203b2df-5ed6-41af-b1a5-341fdbfd92a5" path="/var/lib/kubelet/pods/3203b2df-5ed6-41af-b1a5-341fdbfd92a5/volumes"
Oct 30 14:21:28.139279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.139241    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"]
Oct 30 14:21:28.139807 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.139506    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx" podUID="540d5e5d-ace9-431e-9a4f-982c27948672" containerName="maas-api" containerID="cri-o://93348344fed15e161469df22a267564b9d68db0b814e65ec640c1e06de840569" gracePeriod=30
Oct 30 14:21:28.599364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.599010    4530 generic.go:356] "Generic (PLEG): container finished" podID="540d5e5d-ace9-431e-9a4f-982c27948672" containerID="93348344fed15e161469df22a267564b9d68db0b814e65ec640c1e06de840569" exitCode=0
Oct 30 14:21:28.599364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.599290    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx" event={"ID":"540d5e5d-ace9-431e-9a4f-982c27948672","Type":"ContainerDied","Data":"93348344fed15e161469df22a267564b9d68db0b814e65ec640c1e06de840569"}
Oct 30 14:21:28.654267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.654235    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:21:28.824356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.823453    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540d5e5d-ace9-431e-9a4f-982c27948672-kube-api-access-vw97r\" (UniqueName: \"kubernetes.io/projected/540d5e5d-ace9-431e-9a4f-982c27948672-kube-api-access-vw97r\") pod \"540d5e5d-ace9-431e-9a4f-982c27948672\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") "
Oct 30 14:21:28.824356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.823605    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-maas-api-tls\") pod \"540d5e5d-ace9-431e-9a4f-982c27948672\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") "
Oct 30 14:21:28.827626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.827051    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-metrics-certs\" (UniqueName: \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-metrics-certs\") pod \"540d5e5d-ace9-431e-9a4f-982c27948672\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") "
Oct 30 14:21:28.827626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.827110    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540d5e5d-ace9-431e-9a4f-982c27948672-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540d5e5d-ace9-431e-9a4f-982c27948672-openshift-service-ca\") pod \"540d5e5d-ace9-431e-9a4f-982c27948672\" (UID: \"540d5e5d-ace9-431e-9a4f-982c27948672\") "
Oct 30 14:21:28.828483 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.828449    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540d5e5d-ace9-431e-9a4f-982c27948672-openshift-service-ca" pod "540d5e5d-ace9-431e-9a4f-982c27948672" (UID: "540d5e5d-ace9-431e-9a4f-982c27948672"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:21:28.828960 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.828929    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540d5e5d-ace9-431e-9a4f-982c27948672-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:28.877492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.877233    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-maas-api-tls" pod "540d5e5d-ace9-431e-9a4f-982c27948672" (UID: "540d5e5d-ace9-431e-9a4f-982c27948672"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:21:28.877492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.877412    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540d5e5d-ace9-431e-9a4f-982c27948672-kube-api-access-vw97r" pod "540d5e5d-ace9-431e-9a4f-982c27948672" (UID: "540d5e5d-ace9-431e-9a4f-982c27948672"). InnerVolumeSpecName "kube-api-access-vw97r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:28.884456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.883767    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-metrics-certs" pod "540d5e5d-ace9-431e-9a4f-982c27948672" (UID: "540d5e5d-ace9-431e-9a4f-982c27948672"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:21:28.930162 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.930129    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:28.930162 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.930162    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw97r\" (UniqueName: \"kubernetes.io/projected/540d5e5d-ace9-431e-9a4f-982c27948672-kube-api-access-vw97r\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:28.930411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:28.930174    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540d5e5d-ace9-431e-9a4f-982c27948672-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:29.194288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:29.194200    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"]
Oct 30 14:21:29.194784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:29.194459    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy" containerID="cri-o://a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c" gracePeriod=30
Oct 30 14:21:29.609178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:29.609142    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx" event={"ID":"540d5e5d-ace9-431e-9a4f-982c27948672","Type":"ContainerDied","Data":"284d0ec06782651e04008b61998d2b477c62a1353051773b78f17f7b577105c9"}
Oct 30 14:21:29.609178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:29.609172    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"
Oct 30 14:21:29.609428 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:29.609237    4530 scope.go:122] "RemoveContainer" containerID="93348344fed15e161469df22a267564b9d68db0b814e65ec640c1e06de840569"
Oct 30 14:21:29.676363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:29.676328    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"]
Oct 30 14:21:29.685280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:29.685253    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-972fee-54f8d6b9d6-glxsx"]
Oct 30 14:21:29.971635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:29.961620    4530 patch_prober.go:29] interesting pod/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:21:29.971635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:29.961706    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:21:31.116407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:31.116371    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540d5e5d-ace9-431e-9a4f-982c27948672" path="/var/lib/kubelet/pods/540d5e5d-ace9-431e-9a4f-982c27948672/volumes"
Oct 30 14:21:32.506678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:32.506636    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"]
Oct 30 14:21:32.507212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:32.506903    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" podUID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" containerName="istio-proxy" containerID="cri-o://70e6e394207ef5d567a269c071209769ff53bf8c3fb55718869e506f1d647f39" gracePeriod=30
Oct 30 14:21:34.477908 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.477850    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:21:34.609807 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.609774    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8631d1a-7032-493f-be3d-131ac728904a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8631d1a-7032-493f-be3d-131ac728904a-istio-podinfo\") pod \"f8631d1a-7032-493f-be3d-131ac728904a\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") "
Oct 30 14:21:34.609946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.609833    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-istio-token\" (UniqueName: \"kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-istio-token\") pod \"f8631d1a-7032-493f-be3d-131ac728904a\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") "
Oct 30 14:21:34.609946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.609882    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-socket\") pod \"f8631d1a-7032-493f-be3d-131ac728904a\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") "
Oct 30 14:21:34.609946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.609911    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-kube-api-access-rz59l\" (UniqueName: \"kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-kube-api-access-rz59l\") pod \"f8631d1a-7032-493f-be3d-131ac728904a\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") "
Oct 30 14:21:34.610083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.609964    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-envoy\") pod \"f8631d1a-7032-493f-be3d-131ac728904a\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") "
Oct 30 14:21:34.610083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.610039    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-certs\") pod \"f8631d1a-7032-493f-be3d-131ac728904a\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") "
Oct 30 14:21:34.610194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.610068    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-data\") pod \"f8631d1a-7032-493f-be3d-131ac728904a\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") "
Oct 30 14:21:34.610194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.610139    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8631d1a-7032-493f-be3d-131ac728904a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8631d1a-7032-493f-be3d-131ac728904a-istiod-ca-cert\") pod \"f8631d1a-7032-493f-be3d-131ac728904a\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") "
Oct 30 14:21:34.610315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.610188    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-socket" pod "f8631d1a-7032-493f-be3d-131ac728904a" (UID: "f8631d1a-7032-493f-be3d-131ac728904a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:34.610315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.610291    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-credential-socket\") pod \"f8631d1a-7032-493f-be3d-131ac728904a\" (UID: \"f8631d1a-7032-493f-be3d-131ac728904a\") "
Oct 30 14:21:34.610490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.610461    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-data" pod "f8631d1a-7032-493f-be3d-131ac728904a" (UID: "f8631d1a-7032-493f-be3d-131ac728904a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:34.610576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.610477    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-certs" pod "f8631d1a-7032-493f-be3d-131ac728904a" (UID: "f8631d1a-7032-493f-be3d-131ac728904a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:34.610783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.610755    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-credential-socket" pod "f8631d1a-7032-493f-be3d-131ac728904a" (UID: "f8631d1a-7032-493f-be3d-131ac728904a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:34.610873 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.610803    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8631d1a-7032-493f-be3d-131ac728904a-istiod-ca-cert" pod "f8631d1a-7032-493f-be3d-131ac728904a" (UID: "f8631d1a-7032-493f-be3d-131ac728904a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:21:34.611040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.611023    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:34.611111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.611043    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:34.611111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.611057    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:34.611111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.611070    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8631d1a-7032-493f-be3d-131ac728904a-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:34.611111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.611081    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:34.612256 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.612232    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-kube-api-access-rz59l" pod "f8631d1a-7032-493f-be3d-131ac728904a" (UID: "f8631d1a-7032-493f-be3d-131ac728904a"). InnerVolumeSpecName "kube-api-access-rz59l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:34.612477 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.612454    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8631d1a-7032-493f-be3d-131ac728904a-istio-podinfo" pod "f8631d1a-7032-493f-be3d-131ac728904a" (UID: "f8631d1a-7032-493f-be3d-131ac728904a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:21:34.612566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.612528    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-istio-token" pod "f8631d1a-7032-493f-be3d-131ac728904a" (UID: "f8631d1a-7032-493f-be3d-131ac728904a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:34.612728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.612640    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-envoy" pod "f8631d1a-7032-493f-be3d-131ac728904a" (UID: "f8631d1a-7032-493f-be3d-131ac728904a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:34.712116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.712087    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8631d1a-7032-493f-be3d-131ac728904a-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:34.712116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.712115    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:34.712116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.712125    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz59l\" (UniqueName: \"kubernetes.io/projected/f8631d1a-7032-493f-be3d-131ac728904a-kube-api-access-rz59l\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:34.712378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.712138    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8631d1a-7032-493f-be3d-131ac728904a-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:34.740409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.740327    4530 generic.go:356] "Generic (PLEG): container finished" podID="f8631d1a-7032-493f-be3d-131ac728904a" containerID="a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c" exitCode=0
Oct 30 14:21:34.740409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.740389    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"
Oct 30 14:21:34.740622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.740409    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" event={"ID":"f8631d1a-7032-493f-be3d-131ac728904a","Type":"ContainerDied","Data":"a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c"}
Oct 30 14:21:34.740622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.740485    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9" event={"ID":"f8631d1a-7032-493f-be3d-131ac728904a","Type":"ContainerDied","Data":"b6bca2e6b0cc241c6f6276500125b6c3d855dc0edf3e94b6f8166298b7c2655d"}
Oct 30 14:21:34.740622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.740509    4530 scope.go:122] "RemoveContainer" containerID="a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c"
Oct 30 14:21:34.752996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.752978    4530 scope.go:122] "RemoveContainer" containerID="d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2"
Oct 30 14:21:34.763020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.762998    4530 scope.go:122] "RemoveContainer" containerID="a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c"
Oct 30 14:21:34.763298 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:21:34.763278    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c\": container with ID starting with a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c not found: ID does not exist" containerID="a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c"
Oct 30 14:21:34.763360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.763308    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c"} err="failed to get container status \"a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c\": rpc error: code = NotFound desc = could not find container \"a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c\": container with ID starting with a2bb956b8601cbbbfabbe00c30a354cc9561a59c64f9eb8502c315ea00ee097c not found: ID does not exist"
Oct 30 14:21:34.763360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.763333    4530 scope.go:122] "RemoveContainer" containerID="d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2"
Oct 30 14:21:34.763630 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:21:34.763612    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2\": container with ID starting with d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2 not found: ID does not exist" containerID="d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2"
Oct 30 14:21:34.763680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.763636    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2"} err="failed to get container status \"d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2\": rpc error: code = NotFound desc = could not find container \"d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2\": container with ID starting with d384480f6f46d3620c44b596bc7f059a622d1cfff7b502bfaa9da8db020195c2 not found: ID does not exist"
Oct 30 14:21:34.804079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.804058    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"]
Oct 30 14:21:34.810724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:34.810701    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d727dc-openshift-default-d7cb66b4c-dvll9"]
Oct 30 14:21:35.116572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.116534    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8631d1a-7032-493f-be3d-131ac728904a" path="/var/lib/kubelet/pods/f8631d1a-7032-493f-be3d-131ac728904a/volumes"
Oct 30 14:21:35.906245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.905763    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"]
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.906999    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907024    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907044    4530 cpu_manager.go:414] "containerMap: removing container" podUID="259f2601-7ea2-43b0-abeb-73891558a746" containerName="istio-proxy"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907052    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259f2601-7ea2-43b0-abeb-73891558a746" containerName="istio-proxy"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907073    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907080    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907097    4530 cpu_manager.go:414] "containerMap: removing container" podUID="91d82203-dd8e-4e1f-ad76-0403aea6eaca" containerName="main"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907104    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d82203-dd8e-4e1f-ad76-0403aea6eaca" containerName="main"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907121    4530 cpu_manager.go:414] "containerMap: removing container" podUID="3203b2df-5ed6-41af-b1a5-341fdbfd92a5" containerName="maas-api"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907129    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3203b2df-5ed6-41af-b1a5-341fdbfd92a5" containerName="maas-api"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907205    4530 cpu_manager.go:414] "containerMap: removing container" podUID="540d5e5d-ace9-431e-9a4f-982c27948672" containerName="maas-api"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907213    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540d5e5d-ace9-431e-9a4f-982c27948672" containerName="maas-api"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907387    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="540d5e5d-ace9-431e-9a4f-982c27948672" containerName="maas-api"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907404    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907412    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="259f2601-7ea2-43b0-abeb-73891558a746" containerName="istio-proxy"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907421    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8631d1a-7032-493f-be3d-131ac728904a" containerName="istio-proxy"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907427    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d82203-dd8e-4e1f-ad76-0403aea6eaca" containerName="main"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.907433    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="3203b2df-5ed6-41af-b1a5-341fdbfd92a5" containerName="maas-api"
Oct 30 14:21:35.912102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.911646    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:35.928319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.927789    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"]
Oct 30 14:21:35.935706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:35.935030    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ea9ed2-openshift-default-dockercfg-fk8hk\""
Oct 30 14:21:36.043594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.043529    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.043952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.043928    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.047698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.047667    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.061562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.059631    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.061562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.059695    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.061562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.059763    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l942p\" (UniqueName: \"kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-kube-api-access-l942p\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.061562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.060004    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.061562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.060035    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.061562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.060228    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.168457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.168365    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.168732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.168698    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.169086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.168968    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.169395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.169376    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.169579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.169562    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.169827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.169808    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.170057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.170037    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.170207 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.170190    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.186312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.171445    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.186964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.186930    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.187180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.187155    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.187527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.187504    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.188122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.188093    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.210740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.170356    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l942p\" (UniqueName: \"kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-kube-api-access-l942p\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.300036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.295341    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l942p\" (UniqueName: \"kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-kube-api-access-l942p\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.313603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.310484    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.313603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.311401    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.369320 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.369256    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.543575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.543457    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:36.864739 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:21:36.864712    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4db2ee9_dd55_4d4a_89cc_f92c77d26db8.slice/crio-389fc07b153ef468e0eb12b565b74fde075040d110f70f56a128abbd0ada5fb4 WatchSource:0}: Error finding container 389fc07b153ef468e0eb12b565b74fde075040d110f70f56a128abbd0ada5fb4: Status 404 returned error can't find the container with id 389fc07b153ef468e0eb12b565b74fde075040d110f70f56a128abbd0ada5fb4
Oct 30 14:21:36.867428 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.867402    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"]
Oct 30 14:21:36.867786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.867755    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:36.867830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.867820    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:36.867863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:36.867850    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:37.778430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:37.778387    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" event={"ID":"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8","Type":"ContainerStarted","Data":"fe41117d5c88dfbd418f289642ac23ef1824e9ec78a41596d91400ff4a4d5181"}
Oct 30 14:21:37.778869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:37.778459    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" event={"ID":"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8","Type":"ContainerStarted","Data":"389fc07b153ef468e0eb12b565b74fde075040d110f70f56a128abbd0ada5fb4"}
Oct 30 14:21:37.780931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:37.780897    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"]
Oct 30 14:21:37.781340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:37.781306    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy" containerID="cri-o://1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd" gracePeriod=30
Oct 30 14:21:37.817680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:37.813148    4530 generic.go:356] "Generic (PLEG): container finished" podID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" containerID="70e6e394207ef5d567a269c071209769ff53bf8c3fb55718869e506f1d647f39" exitCode=0
Oct 30 14:21:37.817680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:37.813218    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" event={"ID":"194c5fe5-da5e-4798-99c5-63a82e75ec1e","Type":"ContainerDied","Data":"70e6e394207ef5d567a269c071209769ff53bf8c3fb55718869e506f1d647f39"}
Oct 30 14:21:37.829554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:37.829521    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:21:37.847609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:37.847564    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" podStartSLOduration=2.8475362669999997 podStartE2EDuration="2.847536267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:37.846392107 +0000 UTC m=+1962.878291050" watchObservedRunningTime="2026-10-30 14:21:37.847536267 +0000 UTC m=+1962.879435279"
Oct 30 14:21:38.025563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.014289    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-envoy\") pod \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") "
Oct 30 14:21:38.025563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.014407    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-data\") pod \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") "
Oct 30 14:21:38.025563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.014496    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-socket\") pod \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") "
Oct 30 14:21:38.025563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.014655    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-podinfo\") pod \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") "
Oct 30 14:21:38.025563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.014717    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-certs\") pod \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") "
Oct 30 14:21:38.025563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.014797    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-credential-socket\") pod \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") "
Oct 30 14:21:38.025563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.014844    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-token\" (UniqueName: \"kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-token\") pod \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") "
Oct 30 14:21:38.025563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.014906    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-kube-api-access-jlj2p\" (UniqueName: \"kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-kube-api-access-jlj2p\") pod \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") "
Oct 30 14:21:38.025563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.015004    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istiod-ca-cert\") pod \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\" (UID: \"194c5fe5-da5e-4798-99c5-63a82e75ec1e\") "
Oct 30 14:21:38.025563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.023283    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istiod-ca-cert" pod "194c5fe5-da5e-4798-99c5-63a82e75ec1e" (UID: "194c5fe5-da5e-4798-99c5-63a82e75ec1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:21:38.025563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.023562    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-certs" pod "194c5fe5-da5e-4798-99c5-63a82e75ec1e" (UID: "194c5fe5-da5e-4798-99c5-63a82e75ec1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:38.025563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.023753    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-credential-socket" pod "194c5fe5-da5e-4798-99c5-63a82e75ec1e" (UID: "194c5fe5-da5e-4798-99c5-63a82e75ec1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:38.039603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.038616    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-socket" pod "194c5fe5-da5e-4798-99c5-63a82e75ec1e" (UID: "194c5fe5-da5e-4798-99c5-63a82e75ec1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:38.039603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.038706    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-data" pod "194c5fe5-da5e-4798-99c5-63a82e75ec1e" (UID: "194c5fe5-da5e-4798-99c5-63a82e75ec1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:38.063613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.060224    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-podinfo" pod "194c5fe5-da5e-4798-99c5-63a82e75ec1e" (UID: "194c5fe5-da5e-4798-99c5-63a82e75ec1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:21:38.089838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.081838    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-token" pod "194c5fe5-da5e-4798-99c5-63a82e75ec1e" (UID: "194c5fe5-da5e-4798-99c5-63a82e75ec1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:38.089838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.082001    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-envoy" pod "194c5fe5-da5e-4798-99c5-63a82e75ec1e" (UID: "194c5fe5-da5e-4798-99c5-63a82e75ec1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:38.089838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.089646    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-kube-api-access-jlj2p" pod "194c5fe5-da5e-4798-99c5-63a82e75ec1e" (UID: "194c5fe5-da5e-4798-99c5-63a82e75ec1e"). InnerVolumeSpecName "kube-api-access-jlj2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:38.128561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.126602    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:38.128561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.126640    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:38.128561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.126656    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlj2p\" (UniqueName: \"kubernetes.io/projected/194c5fe5-da5e-4798-99c5-63a82e75ec1e-kube-api-access-jlj2p\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:38.128561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.126670    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:38.128561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.126682    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:38.128561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.126694    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:38.128561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.126705    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:38.128561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.126717    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194c5fe5-da5e-4798-99c5-63a82e75ec1e-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:38.128561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.126729    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194c5fe5-da5e-4798-99c5-63a82e75ec1e-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:38.543846 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.543768    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:38.549658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.549634    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:38.825960 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.825930    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"
Oct 30 14:21:38.826471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.825957    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7" event={"ID":"194c5fe5-da5e-4798-99c5-63a82e75ec1e","Type":"ContainerDied","Data":"d37b741cd476de9e7ce194e10c31ee091e9172ac4bec9c7e2831f184a22e6747"}
Oct 30 14:21:38.826471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.826019    4530 scope.go:122] "RemoveContainer" containerID="70e6e394207ef5d567a269c071209769ff53bf8c3fb55718869e506f1d647f39"
Oct 30 14:21:38.826471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.826390    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:38.827489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.827421    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:21:38.889957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.889924    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"]
Oct 30 14:21:38.903899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:38.903874    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-6f7w7"]
Oct 30 14:21:39.118010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:39.117937    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" path="/var/lib/kubelet/pods/194c5fe5-da5e-4798-99c5-63a82e75ec1e/volumes"
Oct 30 14:21:40.469064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:40.469030    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"]
Oct 30 14:21:40.552786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:40.552756    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"]
Oct 30 14:21:41.372868 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:21:41.372828    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:21:43.091180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.091116    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:21:43.186522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.186483    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-envoy\") pod \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") "
Oct 30 14:21:43.186724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.186668    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-token\" (UniqueName: \"kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-token\") pod \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") "
Oct 30 14:21:43.186805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.186769    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-certs\") pod \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") "
Oct 30 14:21:43.186877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.186827    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-kube-api-access-t9n7p\" (UniqueName: \"kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-kube-api-access-t9n7p\") pod \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") "
Oct 30 14:21:43.186877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.186859    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-socket\") pod \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") "
Oct 30 14:21:43.187004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.186954    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-credential-socket\") pod \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") "
Oct 30 14:21:43.187004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.186993    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istiod-ca-cert\") pod \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") "
Oct 30 14:21:43.187110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.187072    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-data\") pod \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") "
Oct 30 14:21:43.187110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.187105    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-podinfo\") pod \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\" (UID: \"af3e9eea-3660-4b16-8cc8-afe897b3b3b2\") "
Oct 30 14:21:43.187638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.187600    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-credential-socket" pod "af3e9eea-3660-4b16-8cc8-afe897b3b3b2" (UID: "af3e9eea-3660-4b16-8cc8-afe897b3b3b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:43.187833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.187814    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:43.187958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.187932    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-socket" pod "af3e9eea-3660-4b16-8cc8-afe897b3b3b2" (UID: "af3e9eea-3660-4b16-8cc8-afe897b3b3b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:43.188216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.188168    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istiod-ca-cert" pod "af3e9eea-3660-4b16-8cc8-afe897b3b3b2" (UID: "af3e9eea-3660-4b16-8cc8-afe897b3b3b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:21:43.188309 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.188258    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-certs" pod "af3e9eea-3660-4b16-8cc8-afe897b3b3b2" (UID: "af3e9eea-3660-4b16-8cc8-afe897b3b3b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:43.188595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.188533    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-data" pod "af3e9eea-3660-4b16-8cc8-afe897b3b3b2" (UID: "af3e9eea-3660-4b16-8cc8-afe897b3b3b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:43.189354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.189318    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-envoy" pod "af3e9eea-3660-4b16-8cc8-afe897b3b3b2" (UID: "af3e9eea-3660-4b16-8cc8-afe897b3b3b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:21:43.189484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.189460    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-kube-api-access-t9n7p" pod "af3e9eea-3660-4b16-8cc8-afe897b3b3b2" (UID: "af3e9eea-3660-4b16-8cc8-afe897b3b3b2"). InnerVolumeSpecName "kube-api-access-t9n7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:43.189529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.189498    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-token" pod "af3e9eea-3660-4b16-8cc8-afe897b3b3b2" (UID: "af3e9eea-3660-4b16-8cc8-afe897b3b3b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:43.190708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.190681    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-podinfo" pod "af3e9eea-3660-4b16-8cc8-afe897b3b3b2" (UID: "af3e9eea-3660-4b16-8cc8-afe897b3b3b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:21:43.288975 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.288944    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:43.288975 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.288971    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:43.288975 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.288979    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:43.289184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.288988    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:43.289184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.288995    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:43.289184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.289003    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9n7p\" (UniqueName: \"kubernetes.io/projected/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-kube-api-access-t9n7p\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:43.289184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.289013    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:43.289184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.289021    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3e9eea-3660-4b16-8cc8-afe897b3b3b2-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:43.874030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.873989    4530 generic.go:356] "Generic (PLEG): container finished" podID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerID="1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd" exitCode=0
Oct 30 14:21:43.874204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.874037    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"
Oct 30 14:21:43.874204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.874070    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" event={"ID":"af3e9eea-3660-4b16-8cc8-afe897b3b3b2","Type":"ContainerDied","Data":"1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd"}
Oct 30 14:21:43.874204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.874129    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p" event={"ID":"af3e9eea-3660-4b16-8cc8-afe897b3b3b2","Type":"ContainerDied","Data":"a775cf788670a4d46e412e971f5e02820f0178ffac184fcac45a3fcabf2912e5"}
Oct 30 14:21:43.874204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.874159    4530 scope.go:122] "RemoveContainer" containerID="1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd"
Oct 30 14:21:43.886727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.886708    4530 scope.go:122] "RemoveContainer" containerID="b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b"
Oct 30 14:21:43.896679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.896654    4530 scope.go:122] "RemoveContainer" containerID="1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd"
Oct 30 14:21:43.897020 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:21:43.896991    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd\": container with ID starting with 1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd not found: ID does not exist" containerID="1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd"
Oct 30 14:21:43.897110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.897032    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd"} err="failed to get container status \"1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd\": rpc error: code = NotFound desc = could not find container \"1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd\": container with ID starting with 1b57c73dffc9a0790016eee0c2a65f4ea7a2680a6f89aaeed3fe989858a2c3bd not found: ID does not exist"
Oct 30 14:21:43.897110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.897050    4530 scope.go:122] "RemoveContainer" containerID="b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b"
Oct 30 14:21:43.897344 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:21:43.897320    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b\": container with ID starting with b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b not found: ID does not exist" containerID="b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b"
Oct 30 14:21:43.897392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:43.897356    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b"} err="failed to get container status \"b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b\": rpc error: code = NotFound desc = could not find container \"b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b\": container with ID starting with b95658be4f79036f8450219f856c64290ab953b16a7c914b85ff8cf8a9fe2e6b not found: ID does not exist"
Oct 30 14:21:44.007572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:44.004938    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"]
Oct 30 14:21:44.013000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:44.012944    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-972fee-openshift-default-85db59bcbf-bs28p"]
Oct 30 14:21:45.116627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.116595    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" path="/var/lib/kubelet/pods/af3e9eea-3660-4b16-8cc8-afe897b3b3b2/volumes"
Oct 30 14:21:45.470064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.469980    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9" podUID="79b00782-6b8b-41d1-badb-a4d9c64b13df" containerName="payload-processing" containerID="cri-o://f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73" gracePeriod=25
Oct 30 14:21:45.553118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.553082    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v" podUID="a617c955-7250-4621-bf43-8ec5419d8434" containerName="payload-pre-processing" containerID="cri-o://25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584" gracePeriod=25
Oct 30 14:21:45.851908 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.851874    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"
Oct 30 14:21:45.892636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.892609    4530 generic.go:356] "Generic (PLEG): container finished" podID="79b00782-6b8b-41d1-badb-a4d9c64b13df" containerID="f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73" exitCode=0
Oct 30 14:21:45.892775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.892657    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"
Oct 30 14:21:45.892775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.892693    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9" event={"ID":"79b00782-6b8b-41d1-badb-a4d9c64b13df","Type":"ContainerDied","Data":"f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73"}
Oct 30 14:21:45.892775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.892772    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9" event={"ID":"79b00782-6b8b-41d1-badb-a4d9c64b13df","Type":"ContainerDied","Data":"a053b14bf43df7dc6f6f0f4b4d0bd3118c36e31851212cb8ca9f5435c4283416"}
Oct 30 14:21:45.892933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.892796    4530 scope.go:122] "RemoveContainer" containerID="f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73"
Oct 30 14:21:45.893140 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.893122    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"
Oct 30 14:21:45.894268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.894243    4530 generic.go:356] "Generic (PLEG): container finished" podID="a617c955-7250-4621-bf43-8ec5419d8434" containerID="25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584" exitCode=0
Oct 30 14:21:45.894369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.894274    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v" event={"ID":"a617c955-7250-4621-bf43-8ec5419d8434","Type":"ContainerDied","Data":"25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584"}
Oct 30 14:21:45.894369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.894301    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v" event={"ID":"a617c955-7250-4621-bf43-8ec5419d8434","Type":"ContainerDied","Data":"5249c82c92c6228a8b4447e4500f6741e03bc9e91e4d57467d4ddb75f1952815"}
Oct 30 14:21:45.907161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.907140    4530 scope.go:122] "RemoveContainer" containerID="f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73"
Oct 30 14:21:45.907439 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:21:45.907417    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73\": container with ID starting with f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73 not found: ID does not exist" containerID="f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73"
Oct 30 14:21:45.907526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.907451    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73"} err="failed to get container status \"f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73\": rpc error: code = NotFound desc = could not find container \"f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73\": container with ID starting with f78db0e3c0911c9b6c38b7abd3bc42e43abdb40c148e3ef02fe818cfe6c8ab73 not found: ID does not exist"
Oct 30 14:21:45.907526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.907477    4530 scope.go:122] "RemoveContainer" containerID="25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584"
Oct 30 14:21:45.920489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.920463    4530 scope.go:122] "RemoveContainer" containerID="25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584"
Oct 30 14:21:45.924845 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:21:45.924822    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584\": container with ID starting with 25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584 not found: ID does not exist" containerID="25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584"
Oct 30 14:21:45.924944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.924856    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584"} err="failed to get container status \"25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584\": rpc error: code = NotFound desc = could not find container \"25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584\": container with ID starting with 25e5e386081c08873aef6b2902373db412efa5fa4fafbd550b5df3a3ab40a584 not found: ID does not exist"
Oct 30 14:21:45.925058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.925032    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b00782-6b8b-41d1-badb-a4d9c64b13df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b00782-6b8b-41d1-badb-a4d9c64b13df-plugins-config-volume\") pod \"79b00782-6b8b-41d1-badb-a4d9c64b13df\" (UID: \"79b00782-6b8b-41d1-badb-a4d9c64b13df\") "
Oct 30 14:21:45.925251 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.925233    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b00782-6b8b-41d1-badb-a4d9c64b13df-kube-api-access-slk5f\" (UniqueName: \"kubernetes.io/projected/79b00782-6b8b-41d1-badb-a4d9c64b13df-kube-api-access-slk5f\") pod \"79b00782-6b8b-41d1-badb-a4d9c64b13df\" (UID: \"79b00782-6b8b-41d1-badb-a4d9c64b13df\") "
Oct 30 14:21:45.925487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.925456    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b00782-6b8b-41d1-badb-a4d9c64b13df-plugins-config-volume" pod "79b00782-6b8b-41d1-badb-a4d9c64b13df" (UID: "79b00782-6b8b-41d1-badb-a4d9c64b13df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:21:45.926031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.926004    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b00782-6b8b-41d1-badb-a4d9c64b13df-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:45.927766 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:45.927739    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b00782-6b8b-41d1-badb-a4d9c64b13df-kube-api-access-slk5f" pod "79b00782-6b8b-41d1-badb-a4d9c64b13df" (UID: "79b00782-6b8b-41d1-badb-a4d9c64b13df"). InnerVolumeSpecName "kube-api-access-slk5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:46.026873 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:46.026845    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a617c955-7250-4621-bf43-8ec5419d8434-kube-api-access-5z44k\" (UniqueName: \"kubernetes.io/projected/a617c955-7250-4621-bf43-8ec5419d8434-kube-api-access-5z44k\") pod \"a617c955-7250-4621-bf43-8ec5419d8434\" (UID: \"a617c955-7250-4621-bf43-8ec5419d8434\") "
Oct 30 14:21:46.027018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:46.026916    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a617c955-7250-4621-bf43-8ec5419d8434-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a617c955-7250-4621-bf43-8ec5419d8434-plugins-config-volume\") pod \"a617c955-7250-4621-bf43-8ec5419d8434\" (UID: \"a617c955-7250-4621-bf43-8ec5419d8434\") "
Oct 30 14:21:46.027250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:46.027237    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slk5f\" (UniqueName: \"kubernetes.io/projected/79b00782-6b8b-41d1-badb-a4d9c64b13df-kube-api-access-slk5f\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:46.027341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:46.027286    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a617c955-7250-4621-bf43-8ec5419d8434-plugins-config-volume" pod "a617c955-7250-4621-bf43-8ec5419d8434" (UID: "a617c955-7250-4621-bf43-8ec5419d8434"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:21:46.029023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:46.029005    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a617c955-7250-4621-bf43-8ec5419d8434-kube-api-access-5z44k" pod "a617c955-7250-4621-bf43-8ec5419d8434" (UID: "a617c955-7250-4621-bf43-8ec5419d8434"). InnerVolumeSpecName "kube-api-access-5z44k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:46.141257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:46.128782    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z44k\" (UniqueName: \"kubernetes.io/projected/a617c955-7250-4621-bf43-8ec5419d8434-kube-api-access-5z44k\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:46.142162 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:46.141990    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a617c955-7250-4621-bf43-8ec5419d8434-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:46.247709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:46.247669    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"]
Oct 30 14:21:46.254804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:46.254776    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b5ea88-59d675d48b-pwrj9"]
Oct 30 14:21:46.902848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:46.902819    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"
Oct 30 14:21:46.968669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:46.968640    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"]
Oct 30 14:21:46.982214 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:46.982184    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b5ea88-55cc6694f9-5cv2v"]
Oct 30 14:21:47.115979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:47.115947    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b00782-6b8b-41d1-badb-a4d9c64b13df" path="/var/lib/kubelet/pods/79b00782-6b8b-41d1-badb-a4d9c64b13df/volumes"
Oct 30 14:21:47.116373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:47.116359    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a617c955-7250-4621-bf43-8ec5419d8434" path="/var/lib/kubelet/pods/a617c955-7250-4621-bf43-8ec5419d8434/volumes"
Oct 30 14:21:48.721963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:48.721922    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"]
Oct 30 14:21:48.758662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:48.758632    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"]
Oct 30 14:21:50.286720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.286688    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"]
Oct 30 14:21:50.287693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.287666    4530 cpu_manager.go:414] "containerMap: removing container" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy"
Oct 30 14:21:50.287693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.287692    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy"
Oct 30 14:21:50.287928 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.287719    4530 cpu_manager.go:414] "containerMap: removing container" podUID="79b00782-6b8b-41d1-badb-a4d9c64b13df" containerName="payload-processing"
Oct 30 14:21:50.287928 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.287727    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b00782-6b8b-41d1-badb-a4d9c64b13df" containerName="payload-processing"
Oct 30 14:21:50.287928 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.287790    4530 cpu_manager.go:414] "containerMap: removing container" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy"
Oct 30 14:21:50.287928 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.287798    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy"
Oct 30 14:21:50.287928 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.287813    4530 cpu_manager.go:414] "containerMap: removing container" podUID="a617c955-7250-4621-bf43-8ec5419d8434" containerName="payload-pre-processing"
Oct 30 14:21:50.287928 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.287821    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a617c955-7250-4621-bf43-8ec5419d8434" containerName="payload-pre-processing"
Oct 30 14:21:50.287928 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.287847    4530 cpu_manager.go:414] "containerMap: removing container" podUID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" containerName="istio-proxy"
Oct 30 14:21:50.287928 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.287854    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" containerName="istio-proxy"
Oct 30 14:21:50.288362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.288020    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy"
Oct 30 14:21:50.288362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.288034    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="a617c955-7250-4621-bf43-8ec5419d8434" containerName="payload-pre-processing"
Oct 30 14:21:50.288362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.288046    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="194c5fe5-da5e-4798-99c5-63a82e75ec1e" containerName="istio-proxy"
Oct 30 14:21:50.288362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.288057    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3e9eea-3660-4b16-8cc8-afe897b3b3b2" containerName="istio-proxy"
Oct 30 14:21:50.288362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.288068    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b00782-6b8b-41d1-badb-a4d9c64b13df" containerName="payload-processing"
Oct 30 14:21:50.291511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.291489    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.322580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.321152    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"]
Oct 30 14:21:50.326987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.326966    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-648d5f-openshift-default-dockercfg-q9rc4\""
Oct 30 14:21:50.332490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.332456    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-socket\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.332649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.332597    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgf7h\" (UniqueName: \"kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-kube-api-access-mgf7h\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.332751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.332727    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-data\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.332820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.332768    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-credential-socket\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.332881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.332807    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-token\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.332991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.332958    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-envoy\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.333090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.333074    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-certs\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.333169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.333146    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-podinfo\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.333242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.333226    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b91f88-14b5-4321-bd83-5e752955ae3b-istiod-ca-cert\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.455574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.455026    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-certs\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.455574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.455096    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-podinfo\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.455574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.455176    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b91f88-14b5-4321-bd83-5e752955ae3b-istiod-ca-cert\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.455574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.455355    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-socket\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.455574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.455418    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgf7h\" (UniqueName: \"kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-kube-api-access-mgf7h\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.455574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.455507    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-data\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.455574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.455556    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-credential-socket\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.456166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.455598    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-token\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.456166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.455652    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-envoy\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.471561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.461336    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b91f88-14b5-4321-bd83-5e752955ae3b-istiod-ca-cert\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.471561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.461683    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-certs\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.471561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.463167    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-data\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.471561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.463414    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-socket\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.471561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.463967    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-credential-socket\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.484654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.478092    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-envoy\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.497566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.485102    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-podinfo\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.518764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.510353    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgf7h\" (UniqueName: \"kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-kube-api-access-mgf7h\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.536804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.534681    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-token\") pod \"e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") " pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.619566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.617922    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:50.990640 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:21:50.990608    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b91f88_14b5_4321_bd83_5e752955ae3b.slice/crio-765f3027701e9ae2edc354050a1a71432afba4a18d2fc48e321490f282300e7e WatchSource:0}: Error finding container 765f3027701e9ae2edc354050a1a71432afba4a18d2fc48e321490f282300e7e: Status 404 returned error can't find the container with id 765f3027701e9ae2edc354050a1a71432afba4a18d2fc48e321490f282300e7e
Oct 30 14:21:50.992533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.992509    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"]
Oct 30 14:21:50.993441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.993408    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:50.993493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.993476    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:50.993530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:50.993504    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:51.952432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:51.952392    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" event={"ID":"07b91f88-14b5-4321-bd83-5e752955ae3b","Type":"ContainerStarted","Data":"453f189ce1475a9f8e93545e2c7cb3da5fbaf826b768c3649384461a46574c90"}
Oct 30 14:21:51.952869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:51.952463    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" event={"ID":"07b91f88-14b5-4321-bd83-5e752955ae3b","Type":"ContainerStarted","Data":"765f3027701e9ae2edc354050a1a71432afba4a18d2fc48e321490f282300e7e"}
Oct 30 14:21:52.006987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:52.006940    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" podStartSLOduration=2.006925983 podStartE2EDuration="2.006925983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:52.002641534 +0000 UTC m=+1977.034540484" watchObservedRunningTime="2026-10-30 14:21:52.006925983 +0000 UTC m=+1977.038824916"
Oct 30 14:21:52.620519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:52.620377    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:52.626527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:52.626500    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:52.960650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:52.960570    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:52.961605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:52.961587    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:21:53.732672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:53.726485    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl" podUID="cb7a5a16-b15e-4724-88c2-853c833cd9dc" containerName="payload-processing" containerID="cri-o://70a36e7d57a52984c11b26c13e5c40c3bb72b651da37b9427cfdb7205c6d80f1" gracePeriod=25
Oct 30 14:21:53.759882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:53.759836    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9" podUID="94883728-37a4-412f-9330-01fd7d181855" containerName="payload-pre-processing" containerID="cri-o://75130e8bee7f11637b4d4964832b7cca1cf1ba449c990d13251ba8a1c0e4fde9" gracePeriod=25
Oct 30 14:21:53.984470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:53.981869    4530 generic.go:356] "Generic (PLEG): container finished" podID="cb7a5a16-b15e-4724-88c2-853c833cd9dc" containerID="70a36e7d57a52984c11b26c13e5c40c3bb72b651da37b9427cfdb7205c6d80f1" exitCode=0
Oct 30 14:21:53.984470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:53.982095    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl" event={"ID":"cb7a5a16-b15e-4724-88c2-853c833cd9dc","Type":"ContainerDied","Data":"70a36e7d57a52984c11b26c13e5c40c3bb72b651da37b9427cfdb7205c6d80f1"}
Oct 30 14:21:53.989157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:53.989102    4530 generic.go:356] "Generic (PLEG): container finished" podID="94883728-37a4-412f-9330-01fd7d181855" containerID="75130e8bee7f11637b4d4964832b7cca1cf1ba449c990d13251ba8a1c0e4fde9" exitCode=0
Oct 30 14:21:53.989492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:53.989463    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9" event={"ID":"94883728-37a4-412f-9330-01fd7d181855","Type":"ContainerDied","Data":"75130e8bee7f11637b4d4964832b7cca1cf1ba449c990d13251ba8a1c0e4fde9"}
Oct 30 14:21:54.046889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.046859    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"
Oct 30 14:21:54.120030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.120001    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"
Oct 30 14:21:54.155836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.155811    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb7a5a16-b15e-4724-88c2-853c833cd9dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7a5a16-b15e-4724-88c2-853c833cd9dc-plugins-config-volume\") pod \"cb7a5a16-b15e-4724-88c2-853c833cd9dc\" (UID: \"cb7a5a16-b15e-4724-88c2-853c833cd9dc\") "
Oct 30 14:21:54.155997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.155911    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7a5a16-b15e-4724-88c2-853c833cd9dc-kube-api-access-jtfkf\" (UniqueName: \"kubernetes.io/projected/cb7a5a16-b15e-4724-88c2-853c833cd9dc-kube-api-access-jtfkf\") pod \"cb7a5a16-b15e-4724-88c2-853c833cd9dc\" (UID: \"cb7a5a16-b15e-4724-88c2-853c833cd9dc\") "
Oct 30 14:21:54.156242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.156208    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7a5a16-b15e-4724-88c2-853c833cd9dc-plugins-config-volume" pod "cb7a5a16-b15e-4724-88c2-853c833cd9dc" (UID: "cb7a5a16-b15e-4724-88c2-853c833cd9dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:21:54.156787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.156763    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb7a5a16-b15e-4724-88c2-853c833cd9dc-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:54.158103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.158083    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7a5a16-b15e-4724-88c2-853c833cd9dc-kube-api-access-jtfkf" pod "cb7a5a16-b15e-4724-88c2-853c833cd9dc" (UID: "cb7a5a16-b15e-4724-88c2-853c833cd9dc"). InnerVolumeSpecName "kube-api-access-jtfkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:54.257580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.257471    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94883728-37a4-412f-9330-01fd7d181855-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94883728-37a4-412f-9330-01fd7d181855-plugins-config-volume\") pod \"94883728-37a4-412f-9330-01fd7d181855\" (UID: \"94883728-37a4-412f-9330-01fd7d181855\") "
Oct 30 14:21:54.257752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.257672    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94883728-37a4-412f-9330-01fd7d181855-kube-api-access-c82gm\" (UniqueName: \"kubernetes.io/projected/94883728-37a4-412f-9330-01fd7d181855-kube-api-access-c82gm\") pod \"94883728-37a4-412f-9330-01fd7d181855\" (UID: \"94883728-37a4-412f-9330-01fd7d181855\") "
Oct 30 14:21:54.257853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.257828    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94883728-37a4-412f-9330-01fd7d181855-plugins-config-volume" pod "94883728-37a4-412f-9330-01fd7d181855" (UID: "94883728-37a4-412f-9330-01fd7d181855"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:21:54.258286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.258271    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94883728-37a4-412f-9330-01fd7d181855-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:54.258361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.258287    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtfkf\" (UniqueName: \"kubernetes.io/projected/cb7a5a16-b15e-4724-88c2-853c833cd9dc-kube-api-access-jtfkf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:54.259747 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.259722    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94883728-37a4-412f-9330-01fd7d181855-kube-api-access-c82gm" pod "94883728-37a4-412f-9330-01fd7d181855" (UID: "94883728-37a4-412f-9330-01fd7d181855"). InnerVolumeSpecName "kube-api-access-c82gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:21:54.360030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.359999    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c82gm\" (UniqueName: \"kubernetes.io/projected/94883728-37a4-412f-9330-01fd7d181855-kube-api-access-c82gm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:21:54.998531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.998495    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"
Oct 30 14:21:54.999044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.998521    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl" event={"ID":"cb7a5a16-b15e-4724-88c2-853c833cd9dc","Type":"ContainerDied","Data":"5c09c4d736a9b4aca8cdfcc9ea45b9015dea5538ae880290cb7a977947488d89"}
Oct 30 14:21:54.999044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:54.998631    4530 scope.go:122] "RemoveContainer" containerID="70a36e7d57a52984c11b26c13e5c40c3bb72b651da37b9427cfdb7205c6d80f1"
Oct 30 14:21:55.000344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.000323    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"
Oct 30 14:21:55.000442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.000377    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9" event={"ID":"94883728-37a4-412f-9330-01fd7d181855","Type":"ContainerDied","Data":"e351d6f60a44259961829d3ca4168a537c53c57862ec30b566e2f1b7f228a98c"}
Oct 30 14:21:55.011986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.011964    4530 scope.go:122] "RemoveContainer" containerID="75130e8bee7f11637b4d4964832b7cca1cf1ba449c990d13251ba8a1c0e4fde9"
Oct 30 14:21:55.062034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.062002    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"]
Oct 30 14:21:55.075870 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.075845    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aab379-54cb856f5-6bvr9"]
Oct 30 14:21:55.118860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.118830    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94883728-37a4-412f-9330-01fd7d181855" path="/var/lib/kubelet/pods/94883728-37a4-412f-9330-01fd7d181855/volumes"
Oct 30 14:21:55.119177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.119162    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"]
Oct 30 14:21:55.119230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.119192    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aab379-5c6c8cf798-x5xnl"]
Oct 30 14:21:55.334947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.334908    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"]
Oct 30 14:21:55.335755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.335732    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cb7a5a16-b15e-4724-88c2-853c833cd9dc" containerName="payload-processing"
Oct 30 14:21:55.335755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.335753    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7a5a16-b15e-4724-88c2-853c833cd9dc" containerName="payload-processing"
Oct 30 14:21:55.335964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.335819    4530 cpu_manager.go:414] "containerMap: removing container" podUID="94883728-37a4-412f-9330-01fd7d181855" containerName="payload-pre-processing"
Oct 30 14:21:55.335964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.335827    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94883728-37a4-412f-9330-01fd7d181855" containerName="payload-pre-processing"
Oct 30 14:21:55.336065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.336004    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="94883728-37a4-412f-9330-01fd7d181855" containerName="payload-pre-processing"
Oct 30 14:21:55.336065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.336023    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7a5a16-b15e-4724-88c2-853c833cd9dc" containerName="payload-processing"
Oct 30 14:21:55.339787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.339766    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.363057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.363032    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"]
Oct 30 14:21:55.478910 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.478878    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmvf\" (UniqueName: \"kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-kube-api-access-gqmvf\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.479106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.478947    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.479106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.479024    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a039c6-aa25-40ac-ba46-8f0c59bef603-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.479106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.479060    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.479106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.479095    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.479283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.479246    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.479323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.479282    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.479323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.479305    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.479441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.479419    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.581284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.581247    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.581460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.581364    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmvf\" (UniqueName: \"kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-kube-api-access-gqmvf\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.581460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.581405    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.581460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.581430    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a039c6-aa25-40ac-ba46-8f0c59bef603-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.581835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.581473    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.581835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.581505    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.581835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.581633    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.581835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.581690    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.581835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.581720    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.581835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.581726    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.582128 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.581876    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.582128 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.581967    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.582128 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.582055    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.582516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.582499    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a039c6-aa25-40ac-ba46-8f0c59bef603-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.583963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.583942    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.584203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.584184    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.613492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.613430    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.613620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.613535    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmvf\" (UniqueName: \"kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-kube-api-access-gqmvf\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:55.653729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:55.653696    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:56.053628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.053590    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"]
Oct 30 14:21:56.057208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.057170    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:56.057331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.057244    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:56.057331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.057275    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:21:56.486364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.486322    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"]
Oct 30 14:21:56.491203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.491178    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"
Oct 30 14:21:56.510643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.510617    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-648d5f-dockercfg-h4c4q\""
Oct 30 14:21:56.511430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.511405    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"]
Oct 30 14:21:56.550683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.549413    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-648d5f\""
Oct 30 14:21:56.622209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.621845    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp2g8\" (UniqueName: \"kubernetes.io/projected/376dbba0-6d14-4351-811a-9d9b82c7b8cc-kube-api-access-gp2g8\") pod \"payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc\" (UID: \"376dbba0-6d14-4351-811a-9d9b82c7b8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"
Oct 30 14:21:56.622209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.621915    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376dbba0-6d14-4351-811a-9d9b82c7b8cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc\" (UID: \"376dbba0-6d14-4351-811a-9d9b82c7b8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"
Oct 30 14:21:56.750773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.739749    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp2g8\" (UniqueName: \"kubernetes.io/projected/376dbba0-6d14-4351-811a-9d9b82c7b8cc-kube-api-access-gp2g8\") pod \"payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc\" (UID: \"376dbba0-6d14-4351-811a-9d9b82c7b8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"
Oct 30 14:21:56.750773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.739815    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376dbba0-6d14-4351-811a-9d9b82c7b8cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc\" (UID: \"376dbba0-6d14-4351-811a-9d9b82c7b8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"
Oct 30 14:21:56.750773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.740858    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376dbba0-6d14-4351-811a-9d9b82c7b8cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc\" (UID: \"376dbba0-6d14-4351-811a-9d9b82c7b8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"
Oct 30 14:21:56.815300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.810667    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp2g8\" (UniqueName: \"kubernetes.io/projected/376dbba0-6d14-4351-811a-9d9b82c7b8cc-kube-api-access-gp2g8\") pod \"payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc\" (UID: \"376dbba0-6d14-4351-811a-9d9b82c7b8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"
Oct 30 14:21:56.815300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:56.815054    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"
Oct 30 14:21:57.078266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:57.078219    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" event={"ID":"23a039c6-aa25-40ac-ba46-8f0c59bef603","Type":"ContainerStarted","Data":"db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69"}
Oct 30 14:21:57.078925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:57.078891    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" event={"ID":"23a039c6-aa25-40ac-ba46-8f0c59bef603","Type":"ContainerStarted","Data":"67203a36c9cdb25dada4273212c861ef5077e4498fab92dc376da086e6061ccc"}
Oct 30 14:21:57.119202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:57.119160    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb7a5a16-b15e-4724-88c2-853c833cd9dc" path="/var/lib/kubelet/pods/cb7a5a16-b15e-4724-88c2-853c833cd9dc/volumes"
Oct 30 14:21:57.134710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:57.134660    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" podStartSLOduration=2.134641914 podStartE2EDuration="2.134641914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:57.134071725 +0000 UTC m=+1982.165970681" watchObservedRunningTime="2026-10-30 14:21:57.134641914 +0000 UTC m=+1982.166540855"
Oct 30 14:21:57.213845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:57.213813    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"]
Oct 30 14:21:57.656565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:57.655719    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 14:21:57.656565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:57.655791    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" podUID="23a039c6-aa25-40ac-ba46-8f0c59bef603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 14:21:57.656565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:57.656121    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:21:58.112423 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:58.112386    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc" event={"ID":"376dbba0-6d14-4351-811a-9d9b82c7b8cc","Type":"ContainerStarted","Data":"36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286"}
Oct 30 14:21:58.112997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:58.112442    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc" event={"ID":"376dbba0-6d14-4351-811a-9d9b82c7b8cc","Type":"ContainerStarted","Data":"7367ce86e6a11226f061709c2cecf9682945a3a99e89d29a04916628f2718cdd"}
Oct 30 14:21:58.112997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:58.112678    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"
Oct 30 14:21:58.159141 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:58.159094    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc" podStartSLOduration=2.159083127 podStartE2EDuration="2.159083127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:58.15819661 +0000 UTC m=+1983.190095548" watchObservedRunningTime="2026-10-30 14:21:58.159083127 +0000 UTC m=+1983.190982065"
Oct 30 14:21:58.654299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:58.654270    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 14:21:58.654477 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:58.654320    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" podUID="23a039c6-aa25-40ac-ba46-8f0c59bef603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 14:21:59.654263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:59.654230    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 14:21:59.654703 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:59.654281    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" podUID="23a039c6-aa25-40ac-ba46-8f0c59bef603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 14:21:59.717301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:59.715853    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"]
Oct 30 14:21:59.717301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:21:59.716070    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb" podUID="b3091155-b159-4764-a9e2-8668aeeb286f" containerName="maas-api" containerID="cri-o://b75cd4b86089e5c79419ac68c04d034cc063587f63e79a5ffa20061426d9bc1d" gracePeriod=30
Oct 30 14:22:00.367118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.366653    4530 generic.go:356] "Generic (PLEG): container finished" podID="b3091155-b159-4764-a9e2-8668aeeb286f" containerID="b75cd4b86089e5c79419ac68c04d034cc063587f63e79a5ffa20061426d9bc1d" exitCode=0
Oct 30 14:22:00.367118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.366982    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb" event={"ID":"b3091155-b159-4764-a9e2-8668aeeb286f","Type":"ContainerDied","Data":"b75cd4b86089e5c79419ac68c04d034cc063587f63e79a5ffa20061426d9bc1d"}
Oct 30 14:22:00.369642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.369615    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:22:00.433355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.433306    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3091155-b159-4764-a9e2-8668aeeb286f-kube-api-access-k5kcv\" (UniqueName: \"kubernetes.io/projected/b3091155-b159-4764-a9e2-8668aeeb286f-kube-api-access-k5kcv\") pod \"b3091155-b159-4764-a9e2-8668aeeb286f\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") "
Oct 30 14:22:00.433669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.433649    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-maas-api-tls\") pod \"b3091155-b159-4764-a9e2-8668aeeb286f\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") "
Oct 30 14:22:00.457887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.456865    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-metrics-certs\") pod \"b3091155-b159-4764-a9e2-8668aeeb286f\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") "
Oct 30 14:22:00.471334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.471294    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3091155-b159-4764-a9e2-8668aeeb286f-kube-api-access-k5kcv" pod "b3091155-b159-4764-a9e2-8668aeeb286f" (UID: "b3091155-b159-4764-a9e2-8668aeeb286f"). InnerVolumeSpecName "kube-api-access-k5kcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:22:00.471623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.471599    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3091155-b159-4764-a9e2-8668aeeb286f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3091155-b159-4764-a9e2-8668aeeb286f-openshift-service-ca\") pod \"b3091155-b159-4764-a9e2-8668aeeb286f\" (UID: \"b3091155-b159-4764-a9e2-8668aeeb286f\") "
Oct 30 14:22:00.472927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.472904    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5kcv\" (UniqueName: \"kubernetes.io/projected/b3091155-b159-4764-a9e2-8668aeeb286f-kube-api-access-k5kcv\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:00.473925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.472079    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3091155-b159-4764-a9e2-8668aeeb286f-openshift-service-ca" pod "b3091155-b159-4764-a9e2-8668aeeb286f" (UID: "b3091155-b159-4764-a9e2-8668aeeb286f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:22:00.477727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.477697    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-metrics-certs" pod "b3091155-b159-4764-a9e2-8668aeeb286f" (UID: "b3091155-b159-4764-a9e2-8668aeeb286f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:22:00.478154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.478114    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-maas-api-tls" pod "b3091155-b159-4764-a9e2-8668aeeb286f" (UID: "b3091155-b159-4764-a9e2-8668aeeb286f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:22:00.575055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.575025    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:00.575055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.575054    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3091155-b159-4764-a9e2-8668aeeb286f-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:00.575267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.575064    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3091155-b159-4764-a9e2-8668aeeb286f-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:00.685608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.682836    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 14:22:00.685608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:00.682899    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" podUID="23a039c6-aa25-40ac-ba46-8f0c59bef603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 14:22:01.376333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:01.376293    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb" event={"ID":"b3091155-b159-4764-a9e2-8668aeeb286f","Type":"ContainerDied","Data":"75bb538ba90ecf89c9d224152b336bc1b449ac31a19b548cf10bed9902de1572"}
Oct 30 14:22:01.376333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:01.376327    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"
Oct 30 14:22:01.376592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:01.376367    4530 scope.go:122] "RemoveContainer" containerID="b75cd4b86089e5c79419ac68c04d034cc063587f63e79a5ffa20061426d9bc1d"
Oct 30 14:22:01.449926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:01.449894    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"]
Oct 30 14:22:01.464144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:01.464117    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb"]
Oct 30 14:22:01.654719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:01.654683    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 14:22:01.654912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:01.654741    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" podUID="23a039c6-aa25-40ac-ba46-8f0c59bef603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 14:22:01.768071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:01.767992    4530 scope.go:122] "RemoveContainer" containerID="85574c8bef999895e7935a21b2b6d9cbf6264ce53672a0a3fac185c28ef4ab1b"
Oct 30 14:22:02.654098 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:02.654063    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 14:22:02.654285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:02.654116    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" podUID="23a039c6-aa25-40ac-ba46-8f0c59bef603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 14:22:03.116863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:03.116833    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3091155-b159-4764-a9e2-8668aeeb286f" path="/var/lib/kubelet/pods/b3091155-b159-4764-a9e2-8668aeeb286f/volumes"
Oct 30 14:22:03.266199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:03.266158    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d74808-764fbb749c-tcqgb" podUID="b3091155-b159-4764-a9e2-8668aeeb286f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.178:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:22:03.659534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:03.659506    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:22:03.660015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:03.660000    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:22:03.660785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:03.660761    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:22:06.059014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.058974    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"]
Oct 30 14:22:06.059482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.059242    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t" podUID="332016dc-20a2-43cf-9a2e-c2e33c2f2dde" containerName="maas-api" containerID="cri-o://413b8bb71ea0ea5b6a37d31a03f843a49e9e60f90726bb9ad58ff960e9c8466e" gracePeriod=30
Oct 30 14:22:06.530566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.529882    4530 generic.go:356] "Generic (PLEG): container finished" podID="332016dc-20a2-43cf-9a2e-c2e33c2f2dde" containerID="413b8bb71ea0ea5b6a37d31a03f843a49e9e60f90726bb9ad58ff960e9c8466e" exitCode=0
Oct 30 14:22:06.530566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.530299    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t" event={"ID":"332016dc-20a2-43cf-9a2e-c2e33c2f2dde","Type":"ContainerDied","Data":"413b8bb71ea0ea5b6a37d31a03f843a49e9e60f90726bb9ad58ff960e9c8466e"}
Oct 30 14:22:06.556494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.556463    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:22:06.712613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.707905    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"]
Oct 30 14:22:06.712613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.709038    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b3091155-b159-4764-a9e2-8668aeeb286f" containerName="maas-api"
Oct 30 14:22:06.712613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.709062    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3091155-b159-4764-a9e2-8668aeeb286f" containerName="maas-api"
Oct 30 14:22:06.712613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.709077    4530 cpu_manager.go:414] "containerMap: removing container" podUID="332016dc-20a2-43cf-9a2e-c2e33c2f2dde" containerName="maas-api"
Oct 30 14:22:06.712613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.709085    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332016dc-20a2-43cf-9a2e-c2e33c2f2dde" containerName="maas-api"
Oct 30 14:22:06.712613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.709284    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3091155-b159-4764-a9e2-8668aeeb286f" containerName="maas-api"
Oct 30 14:22:06.712613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.709298    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="332016dc-20a2-43cf-9a2e-c2e33c2f2dde" containerName="maas-api"
Oct 30 14:22:06.713613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.713078    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:06.755629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.753910    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ca02c9-serving-cert\""
Oct 30 14:22:06.766442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.765791    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-kube-api-access-vjrpm\" (UniqueName: \"kubernetes.io/projected/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-kube-api-access-vjrpm\") pod \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") "
Oct 30 14:22:06.766442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.766000    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-metrics-certs\" (UniqueName: \"kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-metrics-certs\") pod \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") "
Oct 30 14:22:06.766442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.766030    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-maas-api-tls\") pod \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") "
Oct 30 14:22:06.766442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.766143    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-openshift-service-ca\") pod \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\" (UID: \"332016dc-20a2-43cf-9a2e-c2e33c2f2dde\") "
Oct 30 14:22:06.766442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.766312    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb051b8-57d4-430e-b039-6abb961d0fb9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:06.766834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.766460    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-metrics-certs\") pod \"maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:06.766834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.766521    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8chgh\" (UniqueName: \"kubernetes.io/projected/2eb051b8-57d4-430e-b039-6abb961d0fb9-kube-api-access-8chgh\") pod \"maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:06.766834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.766616    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-maas-api-tls\") pod \"maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:06.777574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.774363    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-openshift-service-ca" pod "332016dc-20a2-43cf-9a2e-c2e33c2f2dde" (UID: "332016dc-20a2-43cf-9a2e-c2e33c2f2dde"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:22:06.793568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.789159    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"]
Oct 30 14:22:06.839763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.822945    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-maas-api-tls" pod "332016dc-20a2-43cf-9a2e-c2e33c2f2dde" (UID: "332016dc-20a2-43cf-9a2e-c2e33c2f2dde"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:22:06.839763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.826579    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-kube-api-access-vjrpm" pod "332016dc-20a2-43cf-9a2e-c2e33c2f2dde" (UID: "332016dc-20a2-43cf-9a2e-c2e33c2f2dde"). InnerVolumeSpecName "kube-api-access-vjrpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:22:06.839763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.831669    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-metrics-certs" pod "332016dc-20a2-43cf-9a2e-c2e33c2f2dde" (UID: "332016dc-20a2-43cf-9a2e-c2e33c2f2dde"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:22:06.868002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.867965    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-metrics-certs\") pod \"maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:06.868188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.868075    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8chgh\" (UniqueName: \"kubernetes.io/projected/2eb051b8-57d4-430e-b039-6abb961d0fb9-kube-api-access-8chgh\") pod \"maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:06.868188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.868161    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-maas-api-tls\") pod \"maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:06.868333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.868250    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb051b8-57d4-430e-b039-6abb961d0fb9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:06.868468 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.868449    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:06.868536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.868471    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:06.868536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.868485    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:06.868536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.868499    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjrpm\" (UniqueName: \"kubernetes.io/projected/332016dc-20a2-43cf-9a2e-c2e33c2f2dde-kube-api-access-vjrpm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:06.869887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.869859    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb051b8-57d4-430e-b039-6abb961d0fb9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:06.872514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.872483    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-maas-api-tls\") pod \"maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:06.873082 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.873057    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-metrics-certs\") pod \"maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:06.896005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:06.895974    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8chgh\" (UniqueName: \"kubernetes.io/projected/2eb051b8-57d4-430e-b039-6abb961d0fb9-kube-api-access-8chgh\") pod \"maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:07.164658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:07.164632    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:07.475531 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:22:07.475493    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb051b8_57d4_430e_b039_6abb961d0fb9.slice/crio-57e24917b820e8df0fb8c948f16f0e297245367863e4ff13e5f529e0f0c826b6 WatchSource:0}: Error finding container 57e24917b820e8df0fb8c948f16f0e297245367863e4ff13e5f529e0f0c826b6: Status 404 returned error can't find the container with id 57e24917b820e8df0fb8c948f16f0e297245367863e4ff13e5f529e0f0c826b6
Oct 30 14:22:07.478070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:07.478030    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"]
Oct 30 14:22:07.539787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:07.539754    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm" event={"ID":"2eb051b8-57d4-430e-b039-6abb961d0fb9","Type":"ContainerStarted","Data":"57e24917b820e8df0fb8c948f16f0e297245367863e4ff13e5f529e0f0c826b6"}
Oct 30 14:22:07.541343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:07.541310    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t" event={"ID":"332016dc-20a2-43cf-9a2e-c2e33c2f2dde","Type":"ContainerDied","Data":"e6bb851a3f543e106b8902802a27d08b7878178b6072a21f0f5e4463e76078cb"}
Oct 30 14:22:07.541458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:07.541364    4530 scope.go:122] "RemoveContainer" containerID="413b8bb71ea0ea5b6a37d31a03f843a49e9e60f90726bb9ad58ff960e9c8466e"
Oct 30 14:22:07.541458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:07.541327    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"
Oct 30 14:22:07.604991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:07.604962    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"]
Oct 30 14:22:07.618448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:07.618424    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2b206b-698d9d6bbb-5tk5t"]
Oct 30 14:22:08.553881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:08.553846    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm" event={"ID":"2eb051b8-57d4-430e-b039-6abb961d0fb9","Type":"ContainerStarted","Data":"10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c"}
Oct 30 14:22:08.554269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:08.553979    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:08.604659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:08.604619    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm" podStartSLOduration=2.307957467 podStartE2EDuration="2.604605458s" totalImagesPullingTime="296.647991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:08.60351096 +0000 UTC m=+1993.635409898" watchObservedRunningTime="2026-10-30 14:22:08.604605458 +0000 UTC m=+1993.636504422"
Oct 30 14:22:09.131119 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:09.131084    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="332016dc-20a2-43cf-9a2e-c2e33c2f2dde" path="/var/lib/kubelet/pods/332016dc-20a2-43cf-9a2e-c2e33c2f2dde/volumes"
Oct 30 14:22:09.132143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:09.132117    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"
Oct 30 14:22:09.498704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:09.498612    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"]
Oct 30 14:22:09.499257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:09.498908    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy" containerID="cri-o://38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b" gracePeriod=30
Oct 30 14:22:11.368158 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:22:11.368123    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:22:14.385174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:14.385140    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"]
Oct 30 14:22:14.385995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:14.385391    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy" containerID="cri-o://5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330" gracePeriod=30
Oct 30 14:22:14.595565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:14.592694    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:22:14.915289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:14.915234    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:22:15.061074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.061036    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-data\") pod \"8ace7927-a93c-4a75-879d-be6e0b8aac79\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") "
Oct 30 14:22:15.061393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.061366    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ace7927-a93c-4a75-879d-be6e0b8aac79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ace7927-a93c-4a75-879d-be6e0b8aac79-istiod-ca-cert\") pod \"8ace7927-a93c-4a75-879d-be6e0b8aac79\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") "
Oct 30 14:22:15.062044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.062019    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ace7927-a93c-4a75-879d-be6e0b8aac79-istiod-ca-cert" pod "8ace7927-a93c-4a75-879d-be6e0b8aac79" (UID: "8ace7927-a93c-4a75-879d-be6e0b8aac79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:22:15.063864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.062819    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-credential-socket\") pod \"8ace7927-a93c-4a75-879d-be6e0b8aac79\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") "
Oct 30 14:22:15.063864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.062874    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-certs\") pod \"8ace7927-a93c-4a75-879d-be6e0b8aac79\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") "
Oct 30 14:22:15.063864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.062995    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-kube-api-access-xqnc8\" (UniqueName: \"kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-kube-api-access-xqnc8\") pod \"8ace7927-a93c-4a75-879d-be6e0b8aac79\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") "
Oct 30 14:22:15.063864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.063042    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-token\" (UniqueName: \"kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-token\") pod \"8ace7927-a93c-4a75-879d-be6e0b8aac79\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") "
Oct 30 14:22:15.063864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.063077    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-podinfo\") pod \"8ace7927-a93c-4a75-879d-be6e0b8aac79\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") "
Oct 30 14:22:15.063864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.063101    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-socket\") pod \"8ace7927-a93c-4a75-879d-be6e0b8aac79\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") "
Oct 30 14:22:15.063864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.063148    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-envoy\") pod \"8ace7927-a93c-4a75-879d-be6e0b8aac79\" (UID: \"8ace7927-a93c-4a75-879d-be6e0b8aac79\") "
Oct 30 14:22:15.063864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.063634    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-data" pod "8ace7927-a93c-4a75-879d-be6e0b8aac79" (UID: "8ace7927-a93c-4a75-879d-be6e0b8aac79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:15.063864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.063712    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ace7927-a93c-4a75-879d-be6e0b8aac79-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:15.064393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.064264    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-credential-socket" pod "8ace7927-a93c-4a75-879d-be6e0b8aac79" (UID: "8ace7927-a93c-4a75-879d-be6e0b8aac79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:15.064585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.064476    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-certs" pod "8ace7927-a93c-4a75-879d-be6e0b8aac79" (UID: "8ace7927-a93c-4a75-879d-be6e0b8aac79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:15.066122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.066092    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-socket" pod "8ace7927-a93c-4a75-879d-be6e0b8aac79" (UID: "8ace7927-a93c-4a75-879d-be6e0b8aac79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:15.067501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.067467    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-kube-api-access-xqnc8" pod "8ace7927-a93c-4a75-879d-be6e0b8aac79" (UID: "8ace7927-a93c-4a75-879d-be6e0b8aac79"). InnerVolumeSpecName "kube-api-access-xqnc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:22:15.068767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.068729    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-podinfo" pod "8ace7927-a93c-4a75-879d-be6e0b8aac79" (UID: "8ace7927-a93c-4a75-879d-be6e0b8aac79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:22:15.073809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.073740    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-envoy" pod "8ace7927-a93c-4a75-879d-be6e0b8aac79" (UID: "8ace7927-a93c-4a75-879d-be6e0b8aac79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:15.075891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.075852    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-token" pod "8ace7927-a93c-4a75-879d-be6e0b8aac79" (UID: "8ace7927-a93c-4a75-879d-be6e0b8aac79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:22:15.164655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.164629    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqnc8\" (UniqueName: \"kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-kube-api-access-xqnc8\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:15.164655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.164653    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:15.164811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.164664    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:15.164811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.164673    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:15.164811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.164683    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:15.164811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.164691    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:15.164811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.164699    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:15.164811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.164709    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ace7927-a93c-4a75-879d-be6e0b8aac79-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:15.658300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.658269    4530 generic.go:356] "Generic (PLEG): container finished" podID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerID="38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b" exitCode=0
Oct 30 14:22:15.658768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.658314    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"
Oct 30 14:22:15.658768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.658321    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" event={"ID":"8ace7927-a93c-4a75-879d-be6e0b8aac79","Type":"ContainerDied","Data":"38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b"}
Oct 30 14:22:15.658768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.658369    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7" event={"ID":"8ace7927-a93c-4a75-879d-be6e0b8aac79","Type":"ContainerDied","Data":"4786d393688b15ba82dd9318a7ca31254e83ce85706596e9c6a95c36208d88ff"}
Oct 30 14:22:15.658768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.658390    4530 scope.go:122] "RemoveContainer" containerID="38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b"
Oct 30 14:22:15.669475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.669458    4530 scope.go:122] "RemoveContainer" containerID="9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2"
Oct 30 14:22:15.678738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.678722    4530 scope.go:122] "RemoveContainer" containerID="38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b"
Oct 30 14:22:15.679005 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:22:15.678983    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b\": container with ID starting with 38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b not found: ID does not exist" containerID="38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b"
Oct 30 14:22:15.679077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.679011    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b"} err="failed to get container status \"38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b\": rpc error: code = NotFound desc = could not find container \"38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b\": container with ID starting with 38141b882d83b7dd1029ad7c7d5715903561548ec5678428d42f21eae8c7223b not found: ID does not exist"
Oct 30 14:22:15.679077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.679030    4530 scope.go:122] "RemoveContainer" containerID="9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2"
Oct 30 14:22:15.679283 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:22:15.679265    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2\": container with ID starting with 9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2 not found: ID does not exist" containerID="9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2"
Oct 30 14:22:15.679334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.679289    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2"} err="failed to get container status \"9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2\": rpc error: code = NotFound desc = could not find container \"9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2\": container with ID starting with 9100e70c5a35d8246a5d0e37ba4dba9dd005361b9de6a18dcc15d8094f8eabc2 not found: ID does not exist"
Oct 30 14:22:15.736707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.736679    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"]
Oct 30 14:22:15.743011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:15.742987    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d74808-openshift-default-5454f78567-g69q7"]
Oct 30 14:22:17.127882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:17.127848    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" path="/var/lib/kubelet/pods/8ace7927-a93c-4a75-879d-be6e0b8aac79/volumes"
Oct 30 14:22:19.663661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.663602    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:22:19.690295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.690268    4530 generic.go:356] "Generic (PLEG): container finished" podID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerID="5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330" exitCode=0
Oct 30 14:22:19.690419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.690313    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"
Oct 30 14:22:19.690419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.690359    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" event={"ID":"75a4813f-02bc-4368-be3c-dcb62d5273cf","Type":"ContainerDied","Data":"5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330"}
Oct 30 14:22:19.690564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.690422    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v" event={"ID":"75a4813f-02bc-4368-be3c-dcb62d5273cf","Type":"ContainerDied","Data":"2ff31f615c878ddd4c2718fa4c74fc09d02bda5d048b8c02f04036abf6979d66"}
Oct 30 14:22:19.690564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.690454    4530 scope.go:122] "RemoveContainer" containerID="5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330"
Oct 30 14:22:19.701174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.701150    4530 scope.go:122] "RemoveContainer" containerID="5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e"
Oct 30 14:22:19.712653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.712636    4530 scope.go:122] "RemoveContainer" containerID="5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330"
Oct 30 14:22:19.712901 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:22:19.712876    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330\": container with ID starting with 5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330 not found: ID does not exist" containerID="5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330"
Oct 30 14:22:19.712956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.712908    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330"} err="failed to get container status \"5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330\": rpc error: code = NotFound desc = could not find container \"5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330\": container with ID starting with 5b7835b77aba303604d6fa7bbd2c77e06cc4b482d26f15be952804461f157330 not found: ID does not exist"
Oct 30 14:22:19.712956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.712925    4530 scope.go:122] "RemoveContainer" containerID="5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e"
Oct 30 14:22:19.713154 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:22:19.713137    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e\": container with ID starting with 5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e not found: ID does not exist" containerID="5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e"
Oct 30 14:22:19.713199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.713159    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e"} err="failed to get container status \"5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e\": rpc error: code = NotFound desc = could not find container \"5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e\": container with ID starting with 5338d6a9fd6b2fbf2e4cf02ec5fbbbabb52c8bbdf54278e9f4dae3a80f20913e not found: ID does not exist"
Oct 30 14:22:19.749249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.749219    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-envoy\") pod \"75a4813f-02bc-4368-be3c-dcb62d5273cf\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") "
Oct 30 14:22:19.749386 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.749303    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-data\") pod \"75a4813f-02bc-4368-be3c-dcb62d5273cf\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") "
Oct 30 14:22:19.749386 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.749369    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-podinfo\") pod \"75a4813f-02bc-4368-be3c-dcb62d5273cf\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") "
Oct 30 14:22:19.749534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.749504    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-kube-api-access-2gdbs\" (UniqueName: \"kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-kube-api-access-2gdbs\") pod \"75a4813f-02bc-4368-be3c-dcb62d5273cf\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") "
Oct 30 14:22:19.750006 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.749985    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-credential-socket\") pod \"75a4813f-02bc-4368-be3c-dcb62d5273cf\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") "
Oct 30 14:22:19.750587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.750153    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-data" pod "75a4813f-02bc-4368-be3c-dcb62d5273cf" (UID: "75a4813f-02bc-4368-be3c-dcb62d5273cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:19.750587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.750292    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-credential-socket" pod "75a4813f-02bc-4368-be3c-dcb62d5273cf" (UID: "75a4813f-02bc-4368-be3c-dcb62d5273cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:19.750839 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.750821    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-socket\") pod \"75a4813f-02bc-4368-be3c-dcb62d5273cf\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") "
Oct 30 14:22:19.751260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.751147    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-socket" pod "75a4813f-02bc-4368-be3c-dcb62d5273cf" (UID: "75a4813f-02bc-4368-be3c-dcb62d5273cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:19.751756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.751738    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-token\" (UniqueName: \"kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-token\") pod \"75a4813f-02bc-4368-be3c-dcb62d5273cf\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") "
Oct 30 14:22:19.752035 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752012    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a4813f-02bc-4368-be3c-dcb62d5273cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a4813f-02bc-4368-be3c-dcb62d5273cf-istiod-ca-cert\") pod \"75a4813f-02bc-4368-be3c-dcb62d5273cf\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") "
Oct 30 14:22:19.752156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.751827    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-envoy" pod "75a4813f-02bc-4368-be3c-dcb62d5273cf" (UID: "75a4813f-02bc-4368-be3c-dcb62d5273cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:19.752156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.751953    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-kube-api-access-2gdbs" pod "75a4813f-02bc-4368-be3c-dcb62d5273cf" (UID: "75a4813f-02bc-4368-be3c-dcb62d5273cf"). InnerVolumeSpecName "kube-api-access-2gdbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:22:19.752156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752085    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-certs\") pod \"75a4813f-02bc-4368-be3c-dcb62d5273cf\" (UID: \"75a4813f-02bc-4368-be3c-dcb62d5273cf\") "
Oct 30 14:22:19.752156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752004    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-podinfo" pod "75a4813f-02bc-4368-be3c-dcb62d5273cf" (UID: "75a4813f-02bc-4368-be3c-dcb62d5273cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:22:19.752418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752344    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-certs" pod "75a4813f-02bc-4368-be3c-dcb62d5273cf" (UID: "75a4813f-02bc-4368-be3c-dcb62d5273cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:19.752418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752376    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a4813f-02bc-4368-be3c-dcb62d5273cf-istiod-ca-cert" pod "75a4813f-02bc-4368-be3c-dcb62d5273cf" (UID: "75a4813f-02bc-4368-be3c-dcb62d5273cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:22:19.752854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752830    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:19.753024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752857    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:19.753024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752871    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:19.753024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752885    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gdbs\" (UniqueName: \"kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-kube-api-access-2gdbs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:19.753024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752905    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:19.753024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752918    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:19.753024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752934    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a4813f-02bc-4368-be3c-dcb62d5273cf-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:19.753024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.752947    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a4813f-02bc-4368-be3c-dcb62d5273cf-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:19.753608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.753584    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-token" pod "75a4813f-02bc-4368-be3c-dcb62d5273cf" (UID: "75a4813f-02bc-4368-be3c-dcb62d5273cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:22:19.854489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:19.854461    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a4813f-02bc-4368-be3c-dcb62d5273cf-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:20.047748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.047722    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"]
Oct 30 14:22:20.061703 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.061673    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2b206b-openshift-default-6d945cc57c-9wr8v"]
Oct 30 14:22:20.666531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.666498    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"]
Oct 30 14:22:20.667199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.667183    4530 cpu_manager.go:414] "containerMap: removing container" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy"
Oct 30 14:22:20.667278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.667201    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy"
Oct 30 14:22:20.667278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.667224    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy"
Oct 30 14:22:20.667278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.667232    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy"
Oct 30 14:22:20.667278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.667249    4530 cpu_manager.go:414] "containerMap: removing container" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy"
Oct 30 14:22:20.667278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.667258    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy"
Oct 30 14:22:20.667478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.667462    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy"
Oct 30 14:22:20.667478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.667474    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy"
Oct 30 14:22:20.667564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.667487    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy"
Oct 30 14:22:20.671395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.671376    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:20.689622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.689601    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ea9ed2-serving-cert\""
Oct 30 14:22:20.692952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.692931    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"]
Oct 30 14:22:20.726709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.726682    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:22:20.766806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.766781    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae78c3f3-873c-480c-9624-a27660a9f541-openshift-service-ca\") pod \"maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:20.766948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.766817    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-maas-api-tls\") pod \"maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:20.767019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.766997    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-metrics-certs\") pod \"maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:20.767063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.767049    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qx6s\" (UniqueName: \"kubernetes.io/projected/ae78c3f3-873c-480c-9624-a27660a9f541-kube-api-access-6qx6s\") pod \"maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:20.889563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.889054    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae78c3f3-873c-480c-9624-a27660a9f541-openshift-service-ca\") pod \"maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:20.889563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.889124    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-maas-api-tls\") pod \"maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:20.889563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.889320    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-metrics-certs\") pod \"maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:20.889563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.889382    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qx6s\" (UniqueName: \"kubernetes.io/projected/ae78c3f3-873c-480c-9624-a27660a9f541-kube-api-access-6qx6s\") pod \"maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:20.917611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.901787    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae78c3f3-873c-480c-9624-a27660a9f541-openshift-service-ca\") pod \"maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:20.938562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.934916    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-maas-api-tls\") pod \"maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:20.938562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.935819    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-metrics-certs\") pod \"maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:20.976919 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:20.976876    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qx6s\" (UniqueName: \"kubernetes.io/projected/ae78c3f3-873c-480c-9624-a27660a9f541-kube-api-access-6qx6s\") pod \"maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:21.065564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:21.065197    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:21.293565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:21.283002    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" path="/var/lib/kubelet/pods/75a4813f-02bc-4368-be3c-dcb62d5273cf/volumes"
Oct 30 14:22:21.849189 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:22:21.849148    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae78c3f3_873c_480c_9624_a27660a9f541.slice/crio-10a66b7287e5df5293c96c79f2b8aa15c5df3ca2ba24596a0ca4ba15ae52c5a4 WatchSource:0}: Error finding container 10a66b7287e5df5293c96c79f2b8aa15c5df3ca2ba24596a0ca4ba15ae52c5a4: Status 404 returned error can't find the container with id 10a66b7287e5df5293c96c79f2b8aa15c5df3ca2ba24596a0ca4ba15ae52c5a4
Oct 30 14:22:21.850735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:21.850703    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"]
Oct 30 14:22:21.850883 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:21.850864    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:22:22.776492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:22.776449    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd" event={"ID":"ae78c3f3-873c-480c-9624-a27660a9f541","Type":"ContainerStarted","Data":"cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b"}
Oct 30 14:22:22.776704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:22.776520    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:22.776704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:22.776556    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd" event={"ID":"ae78c3f3-873c-480c-9624-a27660a9f541","Type":"ContainerStarted","Data":"10a66b7287e5df5293c96c79f2b8aa15c5df3ca2ba24596a0ca4ba15ae52c5a4"}
Oct 30 14:22:22.827333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:22.827277    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd" podStartSLOduration=2.387801273 podStartE2EDuration="2.827262376s" totalImagesPullingTime="439.461103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:22.823354466 +0000 UTC m=+2007.855253404" watchObservedRunningTime="2026-10-30 14:22:22.827262376 +0000 UTC m=+2007.859161315"
Oct 30 14:22:28.853451 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:28.853412    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:22:34.298216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.298184    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"]
Oct 30 14:22:34.298871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.298856    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy"
Oct 30 14:22:34.298930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.298873    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ace7927-a93c-4a75-879d-be6e0b8aac79" containerName="istio-proxy"
Oct 30 14:22:34.299021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.299011    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a4813f-02bc-4368-be3c-dcb62d5273cf" containerName="istio-proxy"
Oct 30 14:22:34.304145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.304130    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.322210 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.322182    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"]
Oct 30 14:22:34.331182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.331150    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-648d5f-serving-cert\""
Oct 30 14:22:34.350279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.350252    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-metrics-certs\") pod \"maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.350470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.350450    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8jrk\" (UniqueName: \"kubernetes.io/projected/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-kube-api-access-v8jrk\") pod \"maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.350618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.350596    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-maas-api-tls\") pod \"maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.350694 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.350660    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.452156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.452126    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-metrics-certs\") pod \"maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.452333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.452277    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jrk\" (UniqueName: \"kubernetes.io/projected/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-kube-api-access-v8jrk\") pod \"maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.452422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.452356    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-maas-api-tls\") pod \"maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.452422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.452400    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.453161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.453133    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.454799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.454771    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-metrics-certs\") pod \"maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.454905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.454883    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-maas-api-tls\") pod \"maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.479240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.479220    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jrk\" (UniqueName: \"kubernetes.io/projected/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-kube-api-access-v8jrk\") pod \"maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.619614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.618636    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:34.954109 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:22:34.954080    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9446e240_0a9f_4fcd_8f73_0a8ece3aa52f.slice/crio-2e694074b21cafa846883f47f490295e74b9a268cc93d45f9f192d465967d375 WatchSource:0}: Error finding container 2e694074b21cafa846883f47f490295e74b9a268cc93d45f9f192d465967d375: Status 404 returned error can't find the container with id 2e694074b21cafa846883f47f490295e74b9a268cc93d45f9f192d465967d375
Oct 30 14:22:34.956178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:34.956129    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"]
Oct 30 14:22:35.920874 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:35.920837    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf" event={"ID":"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f","Type":"ContainerStarted","Data":"568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1"}
Oct 30 14:22:35.921310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:35.920924    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf" event={"ID":"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f","Type":"ContainerStarted","Data":"2e694074b21cafa846883f47f490295e74b9a268cc93d45f9f192d465967d375"}
Oct 30 14:22:35.921310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:35.921078    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:35.970857 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:35.970810    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf" podStartSLOduration=1.669797433 podStartE2EDuration="1.970796712s" totalImagesPullingTime="300.999279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:35.968604788 +0000 UTC m=+2021.000503728" watchObservedRunningTime="2026-10-30 14:22:35.970796712 +0000 UTC m=+2021.002695679"
Oct 30 14:22:41.368403 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:22:41.368325    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:22:41.933572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:41.933524    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:22:42.354299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:42.354262    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"]
Oct 30 14:22:42.354499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:42.354451    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy" containerID="cri-o://6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29" gracePeriod=30
Oct 30 14:22:47.625750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.625692    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:22:47.665225 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.665195    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"]
Oct 30 14:22:47.666479 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.666444    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx" podUID="d1ef98d5-ba45-41aa-bbdf-4dca262731f9" containerName="maas-api" containerID="cri-o://685dd24c875ceaa5faee1bfcd2b7657f1dda049da71c166355ff2825f347b9f2" gracePeriod=30
Oct 30 14:22:47.750480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.750449    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-data\") pod \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") "
Oct 30 14:22:47.750648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.750497    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-envoy\") pod \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") "
Oct 30 14:22:47.750648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.750526    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-certs\") pod \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") "
Oct 30 14:22:47.750772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.750674    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-token\" (UniqueName: \"kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-token\") pod \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") "
Oct 30 14:22:47.750772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.750726    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istiod-ca-cert\") pod \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") "
Oct 30 14:22:47.750772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.750757    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-podinfo\") pod \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") "
Oct 30 14:22:47.750936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.750798    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-data" pod "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" (UID: "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:47.750936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.750809    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-socket\") pod \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") "
Oct 30 14:22:47.751070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.751051    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-socket" pod "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" (UID: "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:47.751070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.751059    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-credential-socket\") pod \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") "
Oct 30 14:22:47.751194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.751112    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-kube-api-access-hm7ml\" (UniqueName: \"kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-kube-api-access-hm7ml\") pod \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\" (UID: \"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c\") "
Oct 30 14:22:47.751324 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.751306    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istiod-ca-cert" pod "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" (UID: "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:22:47.751650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.751625    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-credential-socket" pod "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" (UID: "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:47.751848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.751829    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:47.751926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.751853    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:47.751926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.751868    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:47.751926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.751882    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:47.752117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.752099    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-certs" pod "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" (UID: "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:47.753305 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.753275    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-envoy" pod "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" (UID: "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:22:47.753425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.753405    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-token" pod "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" (UID: "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:22:47.753781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.753761    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-podinfo" pod "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" (UID: "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:22:47.753896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.753879    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-kube-api-access-hm7ml" pod "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" (UID: "bc093100-4dd1-4d93-ae2f-9bd51f35bb5c"). InnerVolumeSpecName "kube-api-access-hm7ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:22:47.853923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.853824    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:47.853923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.853857    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:47.853923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.853871    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm7ml\" (UniqueName: \"kubernetes.io/projected/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-kube-api-access-hm7ml\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:47.853923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.853886    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:47.853923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:47.853899    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:48.180568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.167823    4530 generic.go:356] "Generic (PLEG): container finished" podID="d1ef98d5-ba45-41aa-bbdf-4dca262731f9" containerID="685dd24c875ceaa5faee1bfcd2b7657f1dda049da71c166355ff2825f347b9f2" exitCode=0
Oct 30 14:22:48.180568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.168045    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx" event={"ID":"d1ef98d5-ba45-41aa-bbdf-4dca262731f9","Type":"ContainerDied","Data":"685dd24c875ceaa5faee1bfcd2b7657f1dda049da71c166355ff2825f347b9f2"}
Oct 30 14:22:48.204576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.184771    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:22:48.204576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.185817    4530 generic.go:356] "Generic (PLEG): container finished" podID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerID="6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29" exitCode=0
Oct 30 14:22:48.204576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.186043    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" event={"ID":"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c","Type":"ContainerDied","Data":"6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29"}
Oct 30 14:22:48.204576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.186102    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z" event={"ID":"bc093100-4dd1-4d93-ae2f-9bd51f35bb5c","Type":"ContainerDied","Data":"b66e25ce2429b32f6801029230429ac75ab6bc559d5cd2a8642b8aaa458f963e"}
Oct 30 14:22:48.204576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.186131    4530 scope.go:122] "RemoveContainer" containerID="6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29"
Oct 30 14:22:48.204576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.186304    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"
Oct 30 14:22:48.305563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.304319    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-kube-api-access-kdvh8\" (UniqueName: \"kubernetes.io/projected/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-kube-api-access-kdvh8\") pod \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") "
Oct 30 14:22:48.305563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.305078    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-maas-api-tls\") pod \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") "
Oct 30 14:22:48.305563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.305525    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-openshift-service-ca\") pod \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") "
Oct 30 14:22:48.306089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.305899    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-openshift-service-ca" pod "d1ef98d5-ba45-41aa-bbdf-4dca262731f9" (UID: "d1ef98d5-ba45-41aa-bbdf-4dca262731f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:22:48.312898 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.312817    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-metrics-certs\") pod \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\" (UID: \"d1ef98d5-ba45-41aa-bbdf-4dca262731f9\") "
Oct 30 14:22:48.316089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.313989    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:48.316651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.316626    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-kube-api-access-kdvh8" pod "d1ef98d5-ba45-41aa-bbdf-4dca262731f9" (UID: "d1ef98d5-ba45-41aa-bbdf-4dca262731f9"). InnerVolumeSpecName "kube-api-access-kdvh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:22:48.317096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.317075    4530 scope.go:122] "RemoveContainer" containerID="6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29"
Oct 30 14:22:48.317376 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.317349    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-maas-api-tls" pod "d1ef98d5-ba45-41aa-bbdf-4dca262731f9" (UID: "d1ef98d5-ba45-41aa-bbdf-4dca262731f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:22:48.318067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.317869    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-metrics-certs" pod "d1ef98d5-ba45-41aa-bbdf-4dca262731f9" (UID: "d1ef98d5-ba45-41aa-bbdf-4dca262731f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:22:48.318067 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:22:48.317960    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29\": container with ID starting with 6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29 not found: ID does not exist" containerID="6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29"
Oct 30 14:22:48.318067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.317998    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29"} err="failed to get container status \"6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29\": rpc error: code = NotFound desc = could not find container \"6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29\": container with ID starting with 6317ab2e3715bda72145a3bb3125b87bb7616012b902a6862e789220898e3c29 not found: ID does not exist"
Oct 30 14:22:48.346069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.346044    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"]
Oct 30 14:22:48.350348 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.350319    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-s5k6z"]
Oct 30 14:22:48.415346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.415320    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdvh8\" (UniqueName: \"kubernetes.io/projected/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-kube-api-access-kdvh8\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:48.415346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.415344    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:48.415496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:48.415355    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ef98d5-ba45-41aa-bbdf-4dca262731f9-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:49.117041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:49.117010    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" path="/var/lib/kubelet/pods/bc093100-4dd1-4d93-ae2f-9bd51f35bb5c/volumes"
Oct 30 14:22:49.195093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:49.195063    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"
Oct 30 14:22:49.195242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:49.195064    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx" event={"ID":"d1ef98d5-ba45-41aa-bbdf-4dca262731f9","Type":"ContainerDied","Data":"11b020ec902c0f33045489c8210650f3df51884ac7c252e90886a1b2d56c684d"}
Oct 30 14:22:49.195242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:49.195228    4530 scope.go:122] "RemoveContainer" containerID="685dd24c875ceaa5faee1bfcd2b7657f1dda049da71c166355ff2825f347b9f2"
Oct 30 14:22:49.258689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:49.258662    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"]
Oct 30 14:22:49.272731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:49.272704    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b5ea88-5f95dc74d-w5mgx"]
Oct 30 14:22:51.115977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:51.115937    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ef98d5-ba45-41aa-bbdf-4dca262731f9" path="/var/lib/kubelet/pods/d1ef98d5-ba45-41aa-bbdf-4dca262731f9/volumes"
Oct 30 14:22:54.454684 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:54.454649    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"]
Oct 30 14:22:54.455200 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:54.454922    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5" podUID="16eb2d85-8e94-49a4-b651-2aef801a540b" containerName="maas-api" containerID="cri-o://b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd" gracePeriod=30
Oct 30 14:22:54.912568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:54.911962    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:22:55.014223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.005843    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16eb2d85-8e94-49a4-b651-2aef801a540b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eb2d85-8e94-49a4-b651-2aef801a540b-openshift-service-ca\") pod \"16eb2d85-8e94-49a4-b651-2aef801a540b\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") "
Oct 30 14:22:55.014223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.005971    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-maas-api-tls\") pod \"16eb2d85-8e94-49a4-b651-2aef801a540b\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") "
Oct 30 14:22:55.014223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.006195    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-metrics-certs\") pod \"16eb2d85-8e94-49a4-b651-2aef801a540b\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") "
Oct 30 14:22:55.014223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.006317    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16eb2d85-8e94-49a4-b651-2aef801a540b-kube-api-access-bz7gx\" (UniqueName: \"kubernetes.io/projected/16eb2d85-8e94-49a4-b651-2aef801a540b-kube-api-access-bz7gx\") pod \"16eb2d85-8e94-49a4-b651-2aef801a540b\" (UID: \"16eb2d85-8e94-49a4-b651-2aef801a540b\") "
Oct 30 14:22:55.014223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.011198    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16eb2d85-8e94-49a4-b651-2aef801a540b-openshift-service-ca" pod "16eb2d85-8e94-49a4-b651-2aef801a540b" (UID: "16eb2d85-8e94-49a4-b651-2aef801a540b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:22:55.032571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.032473    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-metrics-certs" pod "16eb2d85-8e94-49a4-b651-2aef801a540b" (UID: "16eb2d85-8e94-49a4-b651-2aef801a540b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:22:55.060648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.033964    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-maas-api-tls" pod "16eb2d85-8e94-49a4-b651-2aef801a540b" (UID: "16eb2d85-8e94-49a4-b651-2aef801a540b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:22:55.065110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.064164    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16eb2d85-8e94-49a4-b651-2aef801a540b-kube-api-access-bz7gx" pod "16eb2d85-8e94-49a4-b651-2aef801a540b" (UID: "16eb2d85-8e94-49a4-b651-2aef801a540b"). InnerVolumeSpecName "kube-api-access-bz7gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:22:55.117830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.117087    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:55.117830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.117122    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz7gx\" (UniqueName: \"kubernetes.io/projected/16eb2d85-8e94-49a4-b651-2aef801a540b-kube-api-access-bz7gx\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:55.117830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.117138    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eb2d85-8e94-49a4-b651-2aef801a540b-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:55.117830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.117153    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16eb2d85-8e94-49a4-b651-2aef801a540b-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:22:55.237563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.237224    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:22:55.341621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.340531    4530 generic.go:356] "Generic (PLEG): container finished" podID="16eb2d85-8e94-49a4-b651-2aef801a540b" containerID="b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd" exitCode=0
Oct 30 14:22:55.341621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.340757    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5" event={"ID":"16eb2d85-8e94-49a4-b651-2aef801a540b","Type":"ContainerDied","Data":"b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd"}
Oct 30 14:22:55.341621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.340835    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5" event={"ID":"16eb2d85-8e94-49a4-b651-2aef801a540b","Type":"ContainerDied","Data":"3002a0eceaedd81b5fc6b07f939c37d3256bdb23412d80b93e59c383bd6a9a97"}
Oct 30 14:22:55.341621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.340892    4530 scope.go:122] "RemoveContainer" containerID="b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd"
Oct 30 14:22:55.341621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.340767    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"
Oct 30 14:22:55.435759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.435738    4530 scope.go:122] "RemoveContainer" containerID="b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd"
Oct 30 14:22:55.441340 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:22:55.438804    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd\": container with ID starting with b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd not found: ID does not exist" containerID="b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd"
Oct 30 14:22:55.441340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.438851    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd"} err="failed to get container status \"b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd\": rpc error: code = NotFound desc = could not find container \"b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd\": container with ID starting with b43b9bc437585edbbe0e8cb3501cfda7372400651af3015b37d1d5319c4a8ffd not found: ID does not exist"
Oct 30 14:22:55.459572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.456817    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"]
Oct 30 14:22:55.463053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:55.463021    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5"]
Oct 30 14:22:57.137864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:57.137828    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16eb2d85-8e94-49a4-b651-2aef801a540b" path="/var/lib/kubelet/pods/16eb2d85-8e94-49a4-b651-2aef801a540b/volumes"
Oct 30 14:22:57.632594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:22:57.632510    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aab379-6b9f8bdb46-gw8k5" podUID="16eb2d85-8e94-49a4-b651-2aef801a540b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.192:8443/health\": context deadline exceeded"
Oct 30 14:23:03.682706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:03.682660    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"]
Oct 30 14:23:03.683226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:03.682964    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy" containerID="cri-o://7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be" gracePeriod=30
Oct 30 14:23:04.974313 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:04.974274    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"]
Oct 30 14:23:04.975317 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:04.975299    4530 cpu_manager.go:414] "containerMap: removing container" podUID="16eb2d85-8e94-49a4-b651-2aef801a540b" containerName="maas-api"
Oct 30 14:23:04.975374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:04.975320    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16eb2d85-8e94-49a4-b651-2aef801a540b" containerName="maas-api"
Oct 30 14:23:04.975374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:04.975368    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d1ef98d5-ba45-41aa-bbdf-4dca262731f9" containerName="maas-api"
Oct 30 14:23:04.975439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:04.975376    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ef98d5-ba45-41aa-bbdf-4dca262731f9" containerName="maas-api"
Oct 30 14:23:04.975439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:04.975392    4530 cpu_manager.go:414] "containerMap: removing container" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy"
Oct 30 14:23:04.975439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:04.975399    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy"
Oct 30 14:23:04.975536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:04.975530    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ef98d5-ba45-41aa-bbdf-4dca262731f9" containerName="maas-api"
Oct 30 14:23:04.975536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:04.975555    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc093100-4dd1-4d93-ae2f-9bd51f35bb5c" containerName="istio-proxy"
Oct 30 14:23:04.975650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:04.975573    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="16eb2d85-8e94-49a4-b651-2aef801a540b" containerName="maas-api"
Oct 30 14:23:04.981842 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:04.981822    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.005505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.005477    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"]
Oct 30 14:23:05.008030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.007985    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d15602-openshift-default-dockercfg-bws4s\""
Oct 30 14:23:05.133563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.103941    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-socket\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.133563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.104056    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-podinfo\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.133563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.104112    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-data\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.133563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.104155    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-envoy\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.133563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.104284    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89261fc0-4ce9-4d2a-b886-f730e9424523-istiod-ca-cert\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.133563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.104347    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-certs\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.133563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.104440    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-credential-socket\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.133563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.104502    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-token\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.133563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.104534    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrth2\" (UniqueName: \"kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-kube-api-access-zrth2\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.274853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.274733    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-certs\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.294507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.278161    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-certs\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.294507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.290854    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-credential-socket\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.294507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.281399    4530 patch_prober.go:29] interesting pod/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:23:05.294507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.290926    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-token\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.294507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.290950    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:23:05.294507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.290961    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrth2\" (UniqueName: \"kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-kube-api-access-zrth2\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.294507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.291060    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-socket\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.294507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.291505    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-credential-socket\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.294507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.294457    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-socket\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.298996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.298969    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-podinfo\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.299123 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.299036    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-data\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.299123 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.299077    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-envoy\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.299579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.299222    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89261fc0-4ce9-4d2a-b886-f730e9424523-istiod-ca-cert\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.299579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.299507    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-data\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.299940 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.299920    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89261fc0-4ce9-4d2a-b886-f730e9424523-istiod-ca-cert\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.305105 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.305078    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-envoy\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.315019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.314803    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-podinfo\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.320897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.320748    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrth2\" (UniqueName: \"kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-kube-api-access-zrth2\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.328149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.328122    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-token\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.635609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.631512    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:05.701513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.701342    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"]
Oct 30 14:23:05.709204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.709173    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.719089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.719061    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"]
Oct 30 14:23:05.729625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.728262    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5a9d7b\"/\"openshift-service-ca.crt\""
Oct 30 14:23:05.729625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.728362    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5a9d7b\"/\"default-dockercfg-wdw27\""
Oct 30 14:23:05.729625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.728423    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5a9d7b\"/\"test-model-5a9d7b-kserve-self-signed-certs\""
Oct 30 14:23:05.729625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.728479    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5a9d7b\"/\"kube-root-ca.crt\""
Oct 30 14:23:05.859650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.858243    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-dshm\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.859650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.858417    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tls-certs\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.859650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.858608    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-model-cache\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.859650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.858667    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tmp-dir\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.859650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.858701    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-home\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.859650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.858749    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59s5l\" (UniqueName: \"kubernetes.io/projected/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-kube-api-access-59s5l\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.984612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.962155    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-model-cache\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.984612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.962258    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tmp-dir\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.984612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.962294    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-home\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.984612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.962336    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59s5l\" (UniqueName: \"kubernetes.io/projected/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-kube-api-access-59s5l\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.984612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.962408    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-dshm\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.984612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.962624    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tls-certs\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.984612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.968151    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-home\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.984612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.968480    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-model-cache\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.984612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.968762    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tmp-dir\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:05.984612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:05.979013    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-dshm\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:06.042587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.031805    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tls-certs\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:06.084409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.083347    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59s5l\" (UniqueName: \"kubernetes.io/projected/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-kube-api-access-59s5l\") pod \"test-model-5a9d7b-kserve-65c86c6dd7-x8nwk\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") " pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:06.123149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.122953    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"]
Oct 30 14:23:06.125867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.125822    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:06.125996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.125908    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:06.125996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.125955    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:06.362595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.358554    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"]
Oct 30 14:23:06.366884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.366855    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.381419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.377787    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:06.415563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.403732    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"]
Oct 30 14:23:06.484798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.484754    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.484981 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.484946    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.485068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.485017    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.485068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.485048    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.485187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.485092    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsznk\" (UniqueName: \"kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-kube-api-access-nsznk\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.485187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.485141    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61c8db3-2589-43d8-85a0-c49eccfe6023-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.485303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.485195    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.485303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.485267    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.485415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.485360    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.584438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.584395    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" event={"ID":"89261fc0-4ce9-4d2a-b886-f730e9424523","Type":"ContainerStarted","Data":"559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69"}
Oct 30 14:23:06.584656 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.584459    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" event={"ID":"89261fc0-4ce9-4d2a-b886-f730e9424523","Type":"ContainerStarted","Data":"a5641656c2ee37abf4e8e742740a8cad8e39932948c25d91469bf416a4af592e"}
Oct 30 14:23:06.586487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.586460    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.586622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.586568    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.586711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.586617    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.586711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.586690    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.586825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.586718    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsznk\" (UniqueName: \"kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-kube-api-access-nsznk\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.586825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.586748    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61c8db3-2589-43d8-85a0-c49eccfe6023-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.586825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.586779    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.586825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.586823    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.587051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.586872    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.587051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.586915    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.588156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.587337    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.588156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.587610    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61c8db3-2589-43d8-85a0-c49eccfe6023-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.588156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.587780    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.588156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.587959    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.589871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.589850    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.590579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.590556    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.620835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.620767    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.622278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.622241    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsznk\" (UniqueName: \"kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-kube-api-access-nsznk\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.636658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.636495    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:06.642367 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.642325    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" podStartSLOduration=2.642313741 podStartE2EDuration="2.642313741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:06.637226762 +0000 UTC m=+2051.669125699" watchObservedRunningTime="2026-10-30 14:23:06.642313741 +0000 UTC m=+2051.674212679"
Oct 30 14:23:06.719882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.719843    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:06.809582 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:23:06.803839    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a29c13d_89ea_45f2_b3fe_137f9db7ecd8.slice/crio-0fbda24ec248f049b3e24f320775a64cf9393c22615ef83466cd8faff5a4f7e5 WatchSource:0}: Error finding container 0fbda24ec248f049b3e24f320775a64cf9393c22615ef83466cd8faff5a4f7e5: Status 404 returned error can't find the container with id 0fbda24ec248f049b3e24f320775a64cf9393c22615ef83466cd8faff5a4f7e5
Oct 30 14:23:06.830852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:06.824493    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"]
Oct 30 14:23:07.138303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.138272    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"]
Oct 30 14:23:07.139232 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:23:07.139203    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61c8db3_2589_43d8_85a0_c49eccfe6023.slice/crio-8d98dd679971b00be4222e1ac8614ff27d5156e4cc5b21c05af4ddd4090752f6 WatchSource:0}: Error finding container 8d98dd679971b00be4222e1ac8614ff27d5156e4cc5b21c05af4ddd4090752f6: Status 404 returned error can't find the container with id 8d98dd679971b00be4222e1ac8614ff27d5156e4cc5b21c05af4ddd4090752f6
Oct 30 14:23:07.143235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.143176    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:07.143337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.143290    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:07.143398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.143337    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:07.467191 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.466067    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"]
Oct 30 14:23:07.467191 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.466334    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy" containerID="cri-o://bc785982c6bb96d790edeebaee3c6fd10d5df8a9e7b76e010b85b09b8062bbfa" gracePeriod=30
Oct 30 14:23:07.726342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.726257    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk" event={"ID":"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8","Type":"ContainerStarted","Data":"7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2"}
Oct 30 14:23:07.726749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.726719    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk" event={"ID":"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8","Type":"ContainerStarted","Data":"0fbda24ec248f049b3e24f320775a64cf9393c22615ef83466cd8faff5a4f7e5"}
Oct 30 14:23:07.733632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.733520    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" event={"ID":"d61c8db3-2589-43d8-85a0-c49eccfe6023","Type":"ContainerStarted","Data":"7efc98a313c8ff23993b4af8dff369cb9f8fbfba491ba05cada609b2b6499f63"}
Oct 30 14:23:07.733632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.733585    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" event={"ID":"d61c8db3-2589-43d8-85a0-c49eccfe6023","Type":"ContainerStarted","Data":"8d98dd679971b00be4222e1ac8614ff27d5156e4cc5b21c05af4ddd4090752f6"}
Oct 30 14:23:07.743862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.743834    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:07.782196 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.778955    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk" podStartSLOduration=2.77893582 podStartE2EDuration="2.77893582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:07.778344937 +0000 UTC m=+2052.810243889" watchObservedRunningTime="2026-10-30 14:23:07.77893582 +0000 UTC m=+2052.810834760"
Oct 30 14:23:07.837815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.837741    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" podStartSLOduration=1.837723435 podStartE2EDuration="1.837723435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:07.830091017 +0000 UTC m=+2052.861989983" watchObservedRunningTime="2026-10-30 14:23:07.837723435 +0000 UTC m=+2052.869622376"
Oct 30 14:23:07.969561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.965807    4530 patch_prober.go:29] interesting pod/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 14:23:07.969561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:07.965873    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 14:23:08.402622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.395634    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:08.721032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.720989    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:08.723121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.722988    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 14:23:08.723121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.723034    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" podUID="d61c8db3-2589-43d8-85a0-c49eccfe6023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 14:23:08.743977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.743948    4530 generic.go:356] "Generic (PLEG): container finished" podID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerID="453f189ce1475a9f8e93545e2c7cb3da5fbaf826b768c3649384461a46574c90" exitCode=0
Oct 30 14:23:08.744159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.744081    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" event={"ID":"07b91f88-14b5-4321-bd83-5e752955ae3b","Type":"ContainerDied","Data":"453f189ce1475a9f8e93545e2c7cb3da5fbaf826b768c3649384461a46574c90"}
Oct 30 14:23:08.744513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.744496    4530 scope.go:122] "RemoveContainer" containerID="453f189ce1475a9f8e93545e2c7cb3da5fbaf826b768c3649384461a46574c90"
Oct 30 14:23:08.747353 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.747159    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:08.747353 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.747253    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:08.747353 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.747303    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:08.747921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.747867    4530 generic.go:356] "Generic (PLEG): container finished" podID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerID="fe41117d5c88dfbd418f289642ac23ef1824e9ec78a41596d91400ff4a4d5181" exitCode=0
Oct 30 14:23:08.748037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.748001    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" event={"ID":"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8","Type":"ContainerDied","Data":"fe41117d5c88dfbd418f289642ac23ef1824e9ec78a41596d91400ff4a4d5181"}
Oct 30 14:23:08.749280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.748419    4530 scope.go:122] "RemoveContainer" containerID="fe41117d5c88dfbd418f289642ac23ef1824e9ec78a41596d91400ff4a4d5181"
Oct 30 14:23:08.749280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.749208    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:08.750634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.750596    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:08.751012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.750892    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:08.751569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.751522    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:08.751738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.751020    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:23:08.756731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.756710    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"]
Oct 30 14:23:08.762747 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.762728    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:08.827970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.827887    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:23:08.839505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.839473    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-socket\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:08.867160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.866629    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-34bcc5-openshift-default-dockercfg-jr926\""
Oct 30 14:23:08.867160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.866975    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-data\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:08.867160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.867044    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsc5q\" (UniqueName: \"kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-kube-api-access-xsc5q\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:08.867160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.867082    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-certs\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:08.868370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.867809    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-envoy\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:08.868370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.867928    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-token\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:08.868370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.868209    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-podinfo\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:08.868370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.868261    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3310a9c6-08cc-46ef-8a4a-27e960c28216-istiod-ca-cert\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:08.868370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.868320    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-credential-socket\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:08.990618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.983460    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-podinfo\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:08.990618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.983557    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3310a9c6-08cc-46ef-8a4a-27e960c28216-istiod-ca-cert\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:08.990618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:08.983609    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-credential-socket\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.000447 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.000413    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-credential-socket\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.000964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.000936    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-socket\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.001175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.001154    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-data\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.001312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.001294    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsc5q\" (UniqueName: \"kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-kube-api-access-xsc5q\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.001435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.001419    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-certs\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.001707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.001687    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-envoy\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.001884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.001866    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-token\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.025567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.003260    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3310a9c6-08cc-46ef-8a4a-27e960c28216-istiod-ca-cert\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.139757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.122122    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-socket\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.139757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.122902    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"]
Oct 30 14:23:09.272609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.269267    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-podinfo\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.293830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.292942    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-token\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.340160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.340126    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-certs\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.341729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.341525    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-data\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.343796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.343773    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-envoy\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.368982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.368950    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsc5q\" (UniqueName: \"kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-kube-api-access-xsc5q\") pod \"e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") " pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.490109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.463421    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:09.664945 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.664911    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:23:09.738693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.737895    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 14:23:09.738693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.737968    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" podUID="d61c8db3-2589-43d8-85a0-c49eccfe6023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 14:23:09.777131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.777096    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-envoy\") pod \"5dd82452-192f-4c0c-b16c-0e2522d41778\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") "
Oct 30 14:23:09.777375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.777359    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-data\") pod \"5dd82452-192f-4c0c-b16c-0e2522d41778\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") "
Oct 30 14:23:09.777599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.777579    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-certs\") pod \"5dd82452-192f-4c0c-b16c-0e2522d41778\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") "
Oct 30 14:23:09.777750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.777737    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-token\" (UniqueName: \"kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-token\") pod \"5dd82452-192f-4c0c-b16c-0e2522d41778\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") "
Oct 30 14:23:09.777880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.777867    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd82452-192f-4c0c-b16c-0e2522d41778-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd82452-192f-4c0c-b16c-0e2522d41778-istiod-ca-cert\") pod \"5dd82452-192f-4c0c-b16c-0e2522d41778\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") "
Oct 30 14:23:09.778061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.778048    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-podinfo\") pod \"5dd82452-192f-4c0c-b16c-0e2522d41778\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") "
Oct 30 14:23:09.778165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.778152    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-credential-socket\") pod \"5dd82452-192f-4c0c-b16c-0e2522d41778\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") "
Oct 30 14:23:09.778490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.778468    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-socket\") pod \"5dd82452-192f-4c0c-b16c-0e2522d41778\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") "
Oct 30 14:23:09.778695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.778676    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-kube-api-access-xt2k6\" (UniqueName: \"kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-kube-api-access-xt2k6\") pod \"5dd82452-192f-4c0c-b16c-0e2522d41778\" (UID: \"5dd82452-192f-4c0c-b16c-0e2522d41778\") "
Oct 30 14:23:09.855555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.854749    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-data" pod "5dd82452-192f-4c0c-b16c-0e2522d41778" (UID: "5dd82452-192f-4c0c-b16c-0e2522d41778"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:09.858172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.858135    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-certs" pod "5dd82452-192f-4c0c-b16c-0e2522d41778" (UID: "5dd82452-192f-4c0c-b16c-0e2522d41778"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:09.881565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.864253    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-credential-socket" pod "5dd82452-192f-4c0c-b16c-0e2522d41778" (UID: "5dd82452-192f-4c0c-b16c-0e2522d41778"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:09.881565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.864469    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-socket" pod "5dd82452-192f-4c0c-b16c-0e2522d41778" (UID: "5dd82452-192f-4c0c-b16c-0e2522d41778"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:09.889033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.882746    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd82452-192f-4c0c-b16c-0e2522d41778-istiod-ca-cert" pod "5dd82452-192f-4c0c-b16c-0e2522d41778" (UID: "5dd82452-192f-4c0c-b16c-0e2522d41778"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:23:09.931685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.931641    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-kube-api-access-xt2k6" pod "5dd82452-192f-4c0c-b16c-0e2522d41778" (UID: "5dd82452-192f-4c0c-b16c-0e2522d41778"). InnerVolumeSpecName "kube-api-access-xt2k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:23:09.933164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.933125    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-token" pod "5dd82452-192f-4c0c-b16c-0e2522d41778" (UID: "5dd82452-192f-4c0c-b16c-0e2522d41778"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:23:09.933729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.933706    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-envoy" pod "5dd82452-192f-4c0c-b16c-0e2522d41778" (UID: "5dd82452-192f-4c0c-b16c-0e2522d41778"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:09.934018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.933978    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-podinfo" pod "5dd82452-192f-4c0c-b16c-0e2522d41778" (UID: "5dd82452-192f-4c0c-b16c-0e2522d41778"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:23:09.936615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.936516    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" event={"ID":"07b91f88-14b5-4321-bd83-5e752955ae3b","Type":"ContainerStarted","Data":"3a79a63f9862f4638c8b440343112685b08245651e01afbff6ec7b16ef81a355"}
Oct 30 14:23:09.939508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.939476    4530 generic.go:356] "Generic (PLEG): container finished" podID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerID="7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be" exitCode=0
Oct 30 14:23:09.939681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.939600    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" event={"ID":"5dd82452-192f-4c0c-b16c-0e2522d41778","Type":"ContainerDied","Data":"7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be"}
Oct 30 14:23:09.939681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.939646    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld" event={"ID":"5dd82452-192f-4c0c-b16c-0e2522d41778","Type":"ContainerDied","Data":"7b548dfc1f533585f2e901db4e0e8581610ffbcc2b92cd77f9a2616a1463e6ce"}
Oct 30 14:23:09.939681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.939675    4530 scope.go:122] "RemoveContainer" containerID="7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be"
Oct 30 14:23:09.939864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.939845    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"
Oct 30 14:23:09.944303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.944236    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" event={"ID":"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8","Type":"ContainerStarted","Data":"4b46e3491636130b863d20b3cdd6fcd6351dd192595fb0cf57b66d3a659b63f1"}
Oct 30 14:23:09.947075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.945760    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:09.947075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.945901    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:09.947075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.946057    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:09.947075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.946078    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt2k6\" (UniqueName: \"kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-kube-api-access-xt2k6\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:09.947075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.946093    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:09.947075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.946109    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:09.947075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.946226    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd82452-192f-4c0c-b16c-0e2522d41778-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:09.947075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.946365    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd82452-192f-4c0c-b16c-0e2522d41778-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:09.947075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.946379    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd82452-192f-4c0c-b16c-0e2522d41778-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:09.958741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.958420    4530 scope.go:122] "RemoveContainer" containerID="30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f"
Oct 30 14:23:09.972040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.971955    4530 scope.go:122] "RemoveContainer" containerID="7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be"
Oct 30 14:23:09.972490 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:23:09.972281    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be\": container with ID starting with 7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be not found: ID does not exist" containerID="7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be"
Oct 30 14:23:09.972490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.972336    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be"} err="failed to get container status \"7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be\": rpc error: code = NotFound desc = could not find container \"7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be\": container with ID starting with 7de9dc98f3498807a9c0cb07117ae206b7cec3be5866d7446d977dfb9e4e05be not found: ID does not exist"
Oct 30 14:23:09.972490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.972359    4530 scope.go:122] "RemoveContainer" containerID="30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f"
Oct 30 14:23:09.972784 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:23:09.972677    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f\": container with ID starting with 30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f not found: ID does not exist" containerID="30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f"
Oct 30 14:23:09.972784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:09.972709    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f"} err="failed to get container status \"30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f\": rpc error: code = NotFound desc = could not find container \"30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f\": container with ID starting with 30726ad11878495dfb5abe81558cd5f19586992d944f2b2115baa56adaf1975f not found: ID does not exist"
Oct 30 14:23:10.119603 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:23:10.119571    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3310a9c6_08cc_46ef_8a4a_27e960c28216.slice/crio-0c74ce2d8df920abe6f2f23f62b39b51ff1bd761512ae1fa49eba7ee29b3edb1 WatchSource:0}: Error finding container 0c74ce2d8df920abe6f2f23f62b39b51ff1bd761512ae1fa49eba7ee29b3edb1: Status 404 returned error can't find the container with id 0c74ce2d8df920abe6f2f23f62b39b51ff1bd761512ae1fa49eba7ee29b3edb1
Oct 30 14:23:10.122917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.122765    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:10.123099 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.123015    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:10.123099 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.123086    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:10.123303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.123281    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"]
Oct 30 14:23:10.312279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.311927    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"]
Oct 30 14:23:10.320283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.320252    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-aab379-openshift-default-69594d6c89-wpdld"]
Oct 30 14:23:10.544255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.544185    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:23:10.545116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.545093    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 14:23:10.545203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.545140    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 14:23:10.620292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.620258    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:23:10.621570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.621515    4530 patch_prober.go:29] interesting pod/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 14:23:10.621696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.621584    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 14:23:10.721567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.720804    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 14:23:10.721567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.720868    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" podUID="d61c8db3-2589-43d8-85a0-c49eccfe6023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 14:23:10.971299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.971262    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" event={"ID":"3310a9c6-08cc-46ef-8a4a-27e960c28216","Type":"ContainerStarted","Data":"601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3"}
Oct 30 14:23:10.971558 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:10.971527    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" event={"ID":"3310a9c6-08cc-46ef-8a4a-27e960c28216","Type":"ContainerStarted","Data":"0c74ce2d8df920abe6f2f23f62b39b51ff1bd761512ae1fa49eba7ee29b3edb1"}
Oct 30 14:23:11.053361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.053253    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" podStartSLOduration=3.053230231 podStartE2EDuration="3.053230231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:11.035074614 +0000 UTC m=+2056.066973550" watchObservedRunningTime="2026-10-30 14:23:11.053230231 +0000 UTC m=+2056.085129172"
Oct 30 14:23:11.174066 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.174025    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" path="/var/lib/kubelet/pods/5dd82452-192f-4c0c-b16c-0e2522d41778/volumes"
Oct 30 14:23:11.276783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.276696    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"]
Oct 30 14:23:11.277475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.277457    4530 cpu_manager.go:414] "containerMap: removing container" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy"
Oct 30 14:23:11.277523 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.277476    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy"
Oct 30 14:23:11.277523 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.277494    4530 cpu_manager.go:414] "containerMap: removing container" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy"
Oct 30 14:23:11.277523 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.277499    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy"
Oct 30 14:23:11.277668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.277655    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy"
Oct 30 14:23:11.277719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.277670    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd82452-192f-4c0c-b16c-0e2522d41778" containerName="istio-proxy"
Oct 30 14:23:11.528926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.528831    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:11.529320 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.529298    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"]
Oct 30 14:23:11.548577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.529518    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"
Oct 30 14:23:11.587808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.587765    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 14:23:11.591525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.591128    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 14:23:11.591779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.590222    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d15602-dockercfg-7tjsj\""
Oct 30 14:23:11.591941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.590119    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d15602\""
Oct 30 14:23:11.627758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.627023    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:11.637669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.637627    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r\" (UID: \"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"
Oct 30 14:23:11.637841 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.637763    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rqhf\" (UniqueName: \"kubernetes.io/projected/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-kube-api-access-7rqhf\") pod \"payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r\" (UID: \"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"
Oct 30 14:23:11.684251 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.684216    4530 patch_prober.go:29] interesting pod/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 14:23:11.684508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.684480    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 14:23:11.731705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.731559    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 14:23:11.731705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.731620    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" podUID="d61c8db3-2589-43d8-85a0-c49eccfe6023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 14:23:11.745574 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:23:11.736768    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:23:11.774717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.756295    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r\" (UID: \"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"
Oct 30 14:23:11.775148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.775119    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rqhf\" (UniqueName: \"kubernetes.io/projected/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-kube-api-access-7rqhf\") pod \"payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r\" (UID: \"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"
Oct 30 14:23:11.775682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.771837    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r\" (UID: \"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"
Oct 30 14:23:11.845951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:11.845920    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rqhf\" (UniqueName: \"kubernetes.io/projected/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-kube-api-access-7rqhf\") pod \"payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r\" (UID: \"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"
Oct 30 14:23:12.036563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:12.019663    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:12.140715 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:12.088333    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"
Oct 30 14:23:12.272248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:12.265761    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:23:12.670781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:12.670517    4530 patch_prober.go:29] interesting pod/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 14:23:12.670781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:12.670599    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 14:23:12.673380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:12.672012    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 14:23:12.673380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:12.672059    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 14:23:12.735854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:12.735818    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 14:23:12.736198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:12.736154    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" podUID="d61c8db3-2589-43d8-85a0-c49eccfe6023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 14:23:13.466654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.440220    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"]
Oct 30 14:23:13.475567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.475512    4530 generic.go:356] "Generic (PLEG): container finished" podID="700073be-6e94-45fe-a2e3-d7505c506c85" containerID="bc785982c6bb96d790edeebaee3c6fd10d5df8a9e7b76e010b85b09b8062bbfa" exitCode=0
Oct 30 14:23:13.475719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.475693    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" event={"ID":"700073be-6e94-45fe-a2e3-d7505c506c85","Type":"ContainerDied","Data":"bc785982c6bb96d790edeebaee3c6fd10d5df8a9e7b76e010b85b09b8062bbfa"}
Oct 30 14:23:13.475777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.475770    4530 scope.go:122] "RemoveContainer" containerID="72161d94197ddf52c4c8f10520677db3836631b7d791f0374d5430c1aa33aa84"
Oct 30 14:23:13.551809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.551777    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 14:23:13.552069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.552041    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 14:23:13.653817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.653347    4530 patch_prober.go:29] interesting pod/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 14:23:13.653817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.653410    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 14:23:13.737296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.736742    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 14:23:13.737296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.736798    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" podUID="d61c8db3-2589-43d8-85a0-c49eccfe6023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 14:23:13.755464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.755105    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:23:13.867523 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.867474    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-certs\") pod \"700073be-6e94-45fe-a2e3-d7505c506c85\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") "
Oct 30 14:23:13.868871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.867826    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-certs" pod "700073be-6e94-45fe-a2e3-d7505c506c85" (UID: "700073be-6e94-45fe-a2e3-d7505c506c85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:13.886222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.886175    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-credential-socket\") pod \"700073be-6e94-45fe-a2e3-d7505c506c85\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") "
Oct 30 14:23:13.886564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.886524    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-kube-api-access-77mbj\" (UniqueName: \"kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-kube-api-access-77mbj\") pod \"700073be-6e94-45fe-a2e3-d7505c506c85\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") "
Oct 30 14:23:13.906067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.887023    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-credential-socket" pod "700073be-6e94-45fe-a2e3-d7505c506c85" (UID: "700073be-6e94-45fe-a2e3-d7505c506c85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:13.906067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.900755    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-data\") pod \"700073be-6e94-45fe-a2e3-d7505c506c85\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") "
Oct 30 14:23:13.906067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.900924    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/700073be-6e94-45fe-a2e3-d7505c506c85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700073be-6e94-45fe-a2e3-d7505c506c85-istio-podinfo\") pod \"700073be-6e94-45fe-a2e3-d7505c506c85\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") "
Oct 30 14:23:13.906067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.900956    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700073be-6e94-45fe-a2e3-d7505c506c85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700073be-6e94-45fe-a2e3-d7505c506c85-istiod-ca-cert\") pod \"700073be-6e94-45fe-a2e3-d7505c506c85\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") "
Oct 30 14:23:13.906067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.900999    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-envoy\") pod \"700073be-6e94-45fe-a2e3-d7505c506c85\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") "
Oct 30 14:23:13.906067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.904674    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-socket\") pod \"700073be-6e94-45fe-a2e3-d7505c506c85\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") "
Oct 30 14:23:13.909490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.909164    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-kube-api-access-77mbj" pod "700073be-6e94-45fe-a2e3-d7505c506c85" (UID: "700073be-6e94-45fe-a2e3-d7505c506c85"). InnerVolumeSpecName "kube-api-access-77mbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:23:13.911904 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.911718    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/700073be-6e94-45fe-a2e3-d7505c506c85-istio-podinfo" pod "700073be-6e94-45fe-a2e3-d7505c506c85" (UID: "700073be-6e94-45fe-a2e3-d7505c506c85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:23:13.912249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.912063    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-data" pod "700073be-6e94-45fe-a2e3-d7505c506c85" (UID: "700073be-6e94-45fe-a2e3-d7505c506c85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:13.912807 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.912491    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700073be-6e94-45fe-a2e3-d7505c506c85-istiod-ca-cert" pod "700073be-6e94-45fe-a2e3-d7505c506c85" (UID: "700073be-6e94-45fe-a2e3-d7505c506c85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:23:13.923671 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.915026    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-envoy" pod "700073be-6e94-45fe-a2e3-d7505c506c85" (UID: "700073be-6e94-45fe-a2e3-d7505c506c85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:13.927506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.927446    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-socket" pod "700073be-6e94-45fe-a2e3-d7505c506c85" (UID: "700073be-6e94-45fe-a2e3-d7505c506c85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:13.927734 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.927701    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-istio-token\" (UniqueName: \"kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-istio-token\") pod \"700073be-6e94-45fe-a2e3-d7505c506c85\" (UID: \"700073be-6e94-45fe-a2e3-d7505c506c85\") "
Oct 30 14:23:13.928844 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.928823    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700073be-6e94-45fe-a2e3-d7505c506c85-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:13.952588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.939586    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700073be-6e94-45fe-a2e3-d7505c506c85-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:13.952588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.939620    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:13.952588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.939635    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:13.952588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.939649    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:13.952588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.939664    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:13.952588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.939682    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77mbj\" (UniqueName: \"kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-kube-api-access-77mbj\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:13.952588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.939696    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700073be-6e94-45fe-a2e3-d7505c506c85-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:13.969573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:13.966201    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-istio-token" pod "700073be-6e94-45fe-a2e3-d7505c506c85" (UID: "700073be-6e94-45fe-a2e3-d7505c506c85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:23:14.041855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.041760    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700073be-6e94-45fe-a2e3-d7505c506c85-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:14.552115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.552079    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 14:23:14.552374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.552342    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 14:23:14.558331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.558301    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc" event={"ID":"700073be-6e94-45fe-a2e3-d7505c506c85","Type":"ContainerDied","Data":"5bb87f2043a2e3515891ae70d4268aea198775b9d9d3c1539c58a33c662a2537"}
Oct 30 14:23:14.558555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.558524    4530 scope.go:122] "RemoveContainer" containerID="bc785982c6bb96d790edeebaee3c6fd10d5df8a9e7b76e010b85b09b8062bbfa"
Oct 30 14:23:14.559236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.558687    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"
Oct 30 14:23:14.564431 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.564370    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r" event={"ID":"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81","Type":"ContainerStarted","Data":"2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb"}
Oct 30 14:23:14.564431 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.564406    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r" event={"ID":"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81","Type":"ContainerStarted","Data":"6adae44558e68628314200fe6db82fff387f7f22f6a7a6ef4409dc952f6d6b4f"}
Oct 30 14:23:14.564618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.564576    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"
Oct 30 14:23:14.630324 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.630193    4530 patch_prober.go:29] interesting pod/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 14:23:14.630648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.630601    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 14:23:14.674511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.674459    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r" podStartSLOduration=3.674446787 podStartE2EDuration="3.674446787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:14.672969245 +0000 UTC m=+2059.704868185" watchObservedRunningTime="2026-10-30 14:23:14.674446787 +0000 UTC m=+2059.706345725"
Oct 30 14:23:14.740226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.740189    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"]
Oct 30 14:23:14.740680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.740526    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:14.741194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.741173    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:14.741890 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.741872    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:23:14.744789 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:14.744766    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b5ea88-openshift-default-845d49979c-hcxzc"]
Oct 30 14:23:15.138225 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.138190    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" path="/var/lib/kubelet/pods/700073be-6e94-45fe-a2e3-d7505c506c85/volumes"
Oct 30 14:23:15.544496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.544317    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 14:23:15.544496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.544367    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 14:23:15.626567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.624936    4530 patch_prober.go:29] interesting pod/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 14:23:15.626567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.625002    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 14:23:15.733681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.733312    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"]
Oct 30 14:23:15.734286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.734259    4530 cpu_manager.go:414] "containerMap: removing container" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy"
Oct 30 14:23:15.734286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.734286    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy"
Oct 30 14:23:15.735020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.734311    4530 cpu_manager.go:414] "containerMap: removing container" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy"
Oct 30 14:23:15.735020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.734317    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy"
Oct 30 14:23:15.735020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.734497    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy"
Oct 30 14:23:15.735020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.734512    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="700073be-6e94-45fe-a2e3-d7505c506c85" containerName="istio-proxy"
Oct 30 14:23:15.738820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.738798    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"
Oct 30 14:23:15.753209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.753171    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"]
Oct 30 14:23:15.759882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.759857    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-34bcc5-dockercfg-66xmq\""
Oct 30 14:23:15.760045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.759894    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-34bcc5\""
Oct 30 14:23:15.767259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.767226    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv\" (UID: \"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"
Oct 30 14:23:15.767384 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:15.767360    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trl9z\" (UniqueName: \"kubernetes.io/projected/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-kube-api-access-trl9z\") pod \"payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv\" (UID: \"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"
Oct 30 14:23:16.050563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.050004    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"]
Oct 30 14:23:16.143372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.143326    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv\" (UID: \"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"
Oct 30 14:23:16.147562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.143755    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trl9z\" (UniqueName: \"kubernetes.io/projected/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-kube-api-access-trl9z\") pod \"payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv\" (UID: \"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"
Oct 30 14:23:16.185157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.171700    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv\" (UID: \"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"
Oct 30 14:23:16.292943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.292905    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"
Oct 30 14:23:16.398267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.371443    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"]
Oct 30 14:23:16.412733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.412706    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:16.413592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.404345    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trl9z\" (UniqueName: \"kubernetes.io/projected/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-kube-api-access-trl9z\") pod \"payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv\" (UID: \"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"
Oct 30 14:23:16.413776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.411643    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8vkf\" (UniqueName: \"kubernetes.io/projected/841ad8f3-2367-45ab-b6ff-5af3b631cdad-kube-api-access-k8vkf\") pod \"payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp\" (UID: \"841ad8f3-2367-45ab-b6ff-5af3b631cdad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"
Oct 30 14:23:16.424761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.424726    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841ad8f3-2367-45ab-b6ff-5af3b631cdad-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp\" (UID: \"841ad8f3-2367-45ab-b6ff-5af3b631cdad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"
Oct 30 14:23:16.541563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.537910    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vkf\" (UniqueName: \"kubernetes.io/projected/841ad8f3-2367-45ab-b6ff-5af3b631cdad-kube-api-access-k8vkf\") pod \"payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp\" (UID: \"841ad8f3-2367-45ab-b6ff-5af3b631cdad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"
Oct 30 14:23:16.541563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.538003    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841ad8f3-2367-45ab-b6ff-5af3b631cdad-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp\" (UID: \"841ad8f3-2367-45ab-b6ff-5af3b631cdad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"
Oct 30 14:23:16.552918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.552869    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:16.554027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.553999    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841ad8f3-2367-45ab-b6ff-5af3b631cdad-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp\" (UID: \"841ad8f3-2367-45ab-b6ff-5af3b631cdad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"
Oct 30 14:23:16.554219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.554026    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 14:23:16.554369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.554347    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 14:23:16.601911 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.600744    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vkf\" (UniqueName: \"kubernetes.io/projected/841ad8f3-2367-45ab-b6ff-5af3b631cdad-kube-api-access-k8vkf\") pod \"payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp\" (UID: \"841ad8f3-2367-45ab-b6ff-5af3b631cdad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"
Oct 30 14:23:16.622166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.621927    4530 patch_prober.go:29] interesting pod/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 14:23:16.622166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.621983    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 14:23:16.657791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.657705    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"
Oct 30 14:23:16.663067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.663046    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:23:16.755433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:16.755235    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"
Oct 30 14:23:17.280093 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:23:17.279104    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841ad8f3_2367_45ab_b6ff_5af3b631cdad.slice/crio-44a68af6c65069c8f11ec4c215c23792c6eaf3ec8c2ee0be05b3cc7804ec2ab7 WatchSource:0}: Error finding container 44a68af6c65069c8f11ec4c215c23792c6eaf3ec8c2ee0be05b3cc7804ec2ab7: Status 404 returned error can't find the container with id 44a68af6c65069c8f11ec4c215c23792c6eaf3ec8c2ee0be05b3cc7804ec2ab7
Oct 30 14:23:17.283307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.283273    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"]
Oct 30 14:23:17.295516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.295457    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"]
Oct 30 14:23:17.299473 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:23:17.299421    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1b9781_ff8a_4981_a7fb_99a91c1d47f6.slice/crio-bd4a0d7791973fae4a92b8a9e8874d20ac707bbc2d1ed08ce3ccc1761f05dda9 WatchSource:0}: Error finding container bd4a0d7791973fae4a92b8a9e8874d20ac707bbc2d1ed08ce3ccc1761f05dda9: Status 404 returned error can't find the container with id bd4a0d7791973fae4a92b8a9e8874d20ac707bbc2d1ed08ce3ccc1761f05dda9
Oct 30 14:23:17.549660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.549631    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:23:17.550146 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.550120    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:23:17.551008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.550986    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:23:17.650943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.650906    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv" event={"ID":"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6","Type":"ContainerStarted","Data":"301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28"}
Oct 30 14:23:17.651159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.650972    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv" event={"ID":"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6","Type":"ContainerStarted","Data":"bd4a0d7791973fae4a92b8a9e8874d20ac707bbc2d1ed08ce3ccc1761f05dda9"}
Oct 30 14:23:17.651159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.651006    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"
Oct 30 14:23:17.653622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.653591    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp" event={"ID":"841ad8f3-2367-45ab-b6ff-5af3b631cdad","Type":"ContainerStarted","Data":"fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c"}
Oct 30 14:23:17.653760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.653650    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp" event={"ID":"841ad8f3-2367-45ab-b6ff-5af3b631cdad","Type":"ContainerStarted","Data":"44a68af6c65069c8f11ec4c215c23792c6eaf3ec8c2ee0be05b3cc7804ec2ab7"}
Oct 30 14:23:17.654337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.654315    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"
Oct 30 14:23:17.704937 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.704884    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv" podStartSLOduration=2.7048574629999997 podStartE2EDuration="2.704857463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:17.697764742 +0000 UTC m=+2062.729663683" watchObservedRunningTime="2026-10-30 14:23:17.704857463 +0000 UTC m=+2062.736756406"
Oct 30 14:23:17.708531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.708498    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:23:17.709199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.709170    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:23:17.709944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.709923    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:23:17.748308 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:17.748263    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp" podStartSLOduration=2.7482516439999998 podStartE2EDuration="2.748251644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:17.745661049 +0000 UTC m=+2062.777560003" watchObservedRunningTime="2026-10-30 14:23:17.748251644 +0000 UTC m=+2062.780150582"
Oct 30 14:23:21.902999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:21.902960    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"]
Oct 30 14:23:21.907677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:21.907652    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:21.933112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:21.933081    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"]
Oct 30 14:23:21.938804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:21.938776    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a9542c-openshift-default-dockercfg-zsj9d\""
Oct 30 14:23:22.002079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.002045    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-certs\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.002079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.002082    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istiod-ca-cert\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.002296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.002119    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-data\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.002337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.002290    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbmp\" (UniqueName: \"kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-kube-api-access-rhbmp\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.002383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.002357    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-podinfo\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.002434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.002406    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-token\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.002473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.002462    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-envoy\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.002658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.002634    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-credential-socket\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.002757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.002677    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-socket\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.106578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.106041    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-data\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.106578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.106280    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbmp\" (UniqueName: \"kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-kube-api-access-rhbmp\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.106578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.106432    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-podinfo\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.106578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.106482    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-token\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.107002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.106632    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-envoy\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.107002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.106839    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-credential-socket\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.107002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.106880    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-socket\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.107188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.107033    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-certs\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.129569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.107427    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istiod-ca-cert\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.129569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.109033    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istiod-ca-cert\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.129569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.109414    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-data\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.129569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.119988    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-credential-socket\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.129569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.126937    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-socket\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.133565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.131142    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-envoy\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.156630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.133719    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-podinfo\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.156630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.138841    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-certs\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.192193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.192158    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-token\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.196574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.194775    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbmp\" (UniqueName: \"kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-kube-api-access-rhbmp\") pod \"e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") " pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.243432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.243395    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:22.701962 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:23:22.701933    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a281b3_e4e7_47ff_83b7_c34d7866ac19.slice/crio-bacc46435d8fdffd753170979a6ea22f6dad13026b41d181370db317c8a7dc00 WatchSource:0}: Error finding container bacc46435d8fdffd753170979a6ea22f6dad13026b41d181370db317c8a7dc00: Status 404 returned error can't find the container with id bacc46435d8fdffd753170979a6ea22f6dad13026b41d181370db317c8a7dc00
Oct 30 14:23:22.704821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.704777    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:22.705081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.704872    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:22.705081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.704901    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"]
Oct 30 14:23:22.705081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.704923    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:22.741343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:22.741316    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" event={"ID":"28a281b3-e4e7-47ff-83b7-c34d7866ac19","Type":"ContainerStarted","Data":"bacc46435d8fdffd753170979a6ea22f6dad13026b41d181370db317c8a7dc00"}
Oct 30 14:23:23.753756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:23.753719    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" event={"ID":"28a281b3-e4e7-47ff-83b7-c34d7866ac19","Type":"ContainerStarted","Data":"3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62"}
Oct 30 14:23:23.820592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:23.820527    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" podStartSLOduration=2.820515217 podStartE2EDuration="2.820515217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:23.818077272 +0000 UTC m=+2068.849976212" watchObservedRunningTime="2026-10-30 14:23:23.820515217 +0000 UTC m=+2068.852414156"
Oct 30 14:23:24.244740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:24.244705    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:24.252142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:24.252113    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:24.764600 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:24.764536    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:24.765448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:24.765430    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:23:26.646705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:26.646670    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"
Oct 30 14:23:28.767752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:28.767717    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"
Oct 30 14:23:29.722325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:29.722292    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"
Oct 30 14:23:34.175038 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.175004    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"]
Oct 30 14:23:34.175638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.175585    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s" podUID="3a91d81b-7562-4463-95ce-b35e8d65a664" containerName="maas-api" containerID="cri-o://c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f" gracePeriod=30
Oct 30 14:23:34.578143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.577776    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:23:34.647635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.645931    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"]
Oct 30 14:23:34.647635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.646183    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr" podUID="f224c0b8-2a3e-4dfc-b299-b9f03b5d118c" containerName="maas-api" containerID="cri-o://f198113e982e906487d31c13b7d6d3a97ec65c43656cbe599e7bf04708b3414a" gracePeriod=30
Oct 30 14:23:34.824838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.768388    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-metrics-certs\") pod \"3a91d81b-7562-4463-95ce-b35e8d65a664\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") "
Oct 30 14:23:34.824838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.803870    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-metrics-certs" pod "3a91d81b-7562-4463-95ce-b35e8d65a664" (UID: "3a91d81b-7562-4463-95ce-b35e8d65a664"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:23:34.829693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.828315    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a91d81b-7562-4463-95ce-b35e8d65a664-openshift-service-ca" pod "3a91d81b-7562-4463-95ce-b35e8d65a664" (UID: "3a91d81b-7562-4463-95ce-b35e8d65a664"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:23:34.829693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.768528    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a91d81b-7562-4463-95ce-b35e8d65a664-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a91d81b-7562-4463-95ce-b35e8d65a664-openshift-service-ca\") pod \"3a91d81b-7562-4463-95ce-b35e8d65a664\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") "
Oct 30 14:23:34.829693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.828666    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a91d81b-7562-4463-95ce-b35e8d65a664-kube-api-access-frxz6\" (UniqueName: \"kubernetes.io/projected/3a91d81b-7562-4463-95ce-b35e8d65a664-kube-api-access-frxz6\") pod \"3a91d81b-7562-4463-95ce-b35e8d65a664\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") "
Oct 30 14:23:34.829693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.828765    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-maas-api-tls\") pod \"3a91d81b-7562-4463-95ce-b35e8d65a664\" (UID: \"3a91d81b-7562-4463-95ce-b35e8d65a664\") "
Oct 30 14:23:34.867902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.867870    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a91d81b-7562-4463-95ce-b35e8d65a664-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:34.869876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:34.869852    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:35.075618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.064462    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a91d81b-7562-4463-95ce-b35e8d65a664-kube-api-access-frxz6" pod "3a91d81b-7562-4463-95ce-b35e8d65a664" (UID: "3a91d81b-7562-4463-95ce-b35e8d65a664"). InnerVolumeSpecName "kube-api-access-frxz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:23:35.215114 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.215059    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-maas-api-tls" pod "3a91d81b-7562-4463-95ce-b35e8d65a664" (UID: "3a91d81b-7562-4463-95ce-b35e8d65a664"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:23:35.261564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.257521    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frxz6\" (UniqueName: \"kubernetes.io/projected/3a91d81b-7562-4463-95ce-b35e8d65a664-kube-api-access-frxz6\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:35.261564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.257657    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a91d81b-7562-4463-95ce-b35e8d65a664-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:35.276836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.273956    4530 generic.go:356] "Generic (PLEG): container finished" podID="3a91d81b-7562-4463-95ce-b35e8d65a664" containerID="c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f" exitCode=0
Oct 30 14:23:35.276836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.274600    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"
Oct 30 14:23:35.283205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.278247    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s" event={"ID":"3a91d81b-7562-4463-95ce-b35e8d65a664","Type":"ContainerDied","Data":"c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f"}
Oct 30 14:23:35.283205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.278297    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s" event={"ID":"3a91d81b-7562-4463-95ce-b35e8d65a664","Type":"ContainerDied","Data":"813e76033e86720ab6981dcefc658ce5db1d3fdadb405f09f7bf440d9ea77d69"}
Oct 30 14:23:35.283205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.278333    4530 scope.go:122] "RemoveContainer" containerID="c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f"
Oct 30 14:23:35.311565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.310223    4530 generic.go:356] "Generic (PLEG): container finished" podID="f224c0b8-2a3e-4dfc-b299-b9f03b5d118c" containerID="f198113e982e906487d31c13b7d6d3a97ec65c43656cbe599e7bf04708b3414a" exitCode=0
Oct 30 14:23:35.311565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.310270    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr" event={"ID":"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c","Type":"ContainerDied","Data":"f198113e982e906487d31c13b7d6d3a97ec65c43656cbe599e7bf04708b3414a"}
Oct 30 14:23:35.333070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.333047    4530 scope.go:122] "RemoveContainer" containerID="c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f"
Oct 30 14:23:35.333403 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:23:35.333374    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f\": container with ID starting with c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f not found: ID does not exist" containerID="c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f"
Oct 30 14:23:35.333479 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.333415    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f"} err="failed to get container status \"c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f\": rpc error: code = NotFound desc = could not find container \"c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f\": container with ID starting with c8dcc52c44343f51ebba8e3445724ffee6b6587a233b17d4c65de5f6bed9bc8f not found: ID does not exist"
Oct 30 14:23:35.401134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.401101    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"]
Oct 30 14:23:35.417388 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.417357    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4b82-7876489f9-5rj4s"]
Oct 30 14:23:35.471259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.471232    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:23:35.561242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.561202    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-metrics-certs\") pod \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") "
Oct 30 14:23:35.570716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.570681    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-maas-api-tls\") pod \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") "
Oct 30 14:23:35.571032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.571005    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-openshift-service-ca\") pod \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") "
Oct 30 14:23:35.582374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.571278    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-kube-api-access-z2ksz\" (UniqueName: \"kubernetes.io/projected/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-kube-api-access-z2ksz\") pod \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\" (UID: \"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c\") "
Oct 30 14:23:35.582374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.581255    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-openshift-service-ca" pod "f224c0b8-2a3e-4dfc-b299-b9f03b5d118c" (UID: "f224c0b8-2a3e-4dfc-b299-b9f03b5d118c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:23:35.586174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.586025    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-kube-api-access-z2ksz" pod "f224c0b8-2a3e-4dfc-b299-b9f03b5d118c" (UID: "f224c0b8-2a3e-4dfc-b299-b9f03b5d118c"). InnerVolumeSpecName "kube-api-access-z2ksz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:23:35.586174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.586105    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-maas-api-tls" pod "f224c0b8-2a3e-4dfc-b299-b9f03b5d118c" (UID: "f224c0b8-2a3e-4dfc-b299-b9f03b5d118c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:23:35.586419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.586397    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-metrics-certs" pod "f224c0b8-2a3e-4dfc-b299-b9f03b5d118c" (UID: "f224c0b8-2a3e-4dfc-b299-b9f03b5d118c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:23:35.683736 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.683701    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:35.683736 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.683731    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:35.683736 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.683741    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:35.683969 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.683751    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2ksz\" (UniqueName: \"kubernetes.io/projected/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c-kube-api-access-z2ksz\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:35.698856 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.698821    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"]
Oct 30 14:23:35.699923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.699903    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f224c0b8-2a3e-4dfc-b299-b9f03b5d118c" containerName="maas-api"
Oct 30 14:23:35.699976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.699926    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f224c0b8-2a3e-4dfc-b299-b9f03b5d118c" containerName="maas-api"
Oct 30 14:23:35.700020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.699979    4530 cpu_manager.go:414] "containerMap: removing container" podUID="3a91d81b-7562-4463-95ce-b35e8d65a664" containerName="maas-api"
Oct 30 14:23:35.700020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.699989    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a91d81b-7562-4463-95ce-b35e8d65a664" containerName="maas-api"
Oct 30 14:23:35.700180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.700168    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a91d81b-7562-4463-95ce-b35e8d65a664" containerName="maas-api"
Oct 30 14:23:35.700216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.700194    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f224c0b8-2a3e-4dfc-b299-b9f03b5d118c" containerName="maas-api"
Oct 30 14:23:35.703660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.703638    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.724796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.724772    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ca02c9\"/\"openshift-service-ca.crt\""
Oct 30 14:23:35.724995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.724858    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ca02c9\"/\"kube-root-ca.crt\""
Oct 30 14:23:35.724995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.724906    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"]
Oct 30 14:23:35.785668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.785638    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-dshm\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.785805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.785759    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6rvx\" (UniqueName: \"kubernetes.io/projected/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-kube-api-access-k6rvx\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.785805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.785793    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-model-cache\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.785957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.785930    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-home\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.786091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.786076    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tmp-dir\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.786133 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.786106    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tls-certs\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.800057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.800038    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ca02c9\"/\"test-model-ca02c9-kserve-self-signed-certs\""
Oct 30 14:23:35.800057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.800041    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ca02c9\"/\"default-dockercfg-fjxdl\""
Oct 30 14:23:35.887018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.886955    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-home\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.887155 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.887067    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tmp-dir\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.887155 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.887094    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tls-certs\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.887287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.887166    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-dshm\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.887287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.887250    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rvx\" (UniqueName: \"kubernetes.io/projected/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-kube-api-access-k6rvx\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.887427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.887399    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-model-cache\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.887491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.887427    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-home\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.887491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.887479    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tmp-dir\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.887755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.887728    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-model-cache\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.889451 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.889429    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-dshm\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.889771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.889754    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tls-certs\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:35.917089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:35.917066    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rvx\" (UniqueName: \"kubernetes.io/projected/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-kube-api-access-k6rvx\") pod \"test-model-ca02c9-kserve-674db4d4dc-mbfz4\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") " pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:36.034571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:36.028852    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:36.319202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:36.319166    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"
Oct 30 14:23:36.319677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:36.319199    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr" event={"ID":"f224c0b8-2a3e-4dfc-b299-b9f03b5d118c","Type":"ContainerDied","Data":"d01f41772abe66edd4e2775838951f75ef79fc8c302d157bec05fe0a31f920e4"}
Oct 30 14:23:36.319677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:36.319261    4530 scope.go:122] "RemoveContainer" containerID="f198113e982e906487d31c13b7d6d3a97ec65c43656cbe599e7bf04708b3414a"
Oct 30 14:23:36.435725 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:36.409451    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"]
Oct 30 14:23:36.435725 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:36.430459    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a2835-f4bf8b77b-8s5wr"]
Oct 30 14:23:36.437899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:36.437471    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"]
Oct 30 14:23:37.116282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:37.116252    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a91d81b-7562-4463-95ce-b35e8d65a664" path="/var/lib/kubelet/pods/3a91d81b-7562-4463-95ce-b35e8d65a664/volumes"
Oct 30 14:23:37.116791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:37.116774    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f224c0b8-2a3e-4dfc-b299-b9f03b5d118c" path="/var/lib/kubelet/pods/f224c0b8-2a3e-4dfc-b299-b9f03b5d118c/volumes"
Oct 30 14:23:37.374554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:37.374436    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4" event={"ID":"809d3ffe-ede2-4b4a-a8ff-38ba22659c28","Type":"ContainerStarted","Data":"eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b"}
Oct 30 14:23:37.374554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:37.374491    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4" event={"ID":"809d3ffe-ede2-4b4a-a8ff-38ba22659c28","Type":"ContainerStarted","Data":"c41df3bf9b30ec1959bc5ccf63711e4d7de4dddedb38f89c5b4df229a95b802f"}
Oct 30 14:23:37.427860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:37.427788    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4" podStartSLOduration=2.427768349 podStartE2EDuration="2.427768349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:37.424903162 +0000 UTC m=+2082.456802116" watchObservedRunningTime="2026-10-30 14:23:37.427768349 +0000 UTC m=+2082.459667291"
Oct 30 14:23:38.029526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:38.029493    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:41.369511 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:23:41.369473    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:23:43.225855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:43.225816    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"]
Oct 30 14:23:43.226379 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:43.226087    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerName="istio-proxy" containerID="cri-o://445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26" gracePeriod=30
Oct 30 14:23:44.470721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.470689    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"]
Oct 30 14:23:44.471179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.470904    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" podUID="23a039c6-aa25-40ac-ba46-8f0c59bef603" containerName="istio-proxy" containerID="cri-o://db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69" gracePeriod=30
Oct 30 14:23:44.482002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.481976    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"]
Oct 30 14:23:44.482172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.482155    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy" containerID="cri-o://76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a" gracePeriod=30
Oct 30 14:23:44.555411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.555376    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"]
Oct 30 14:23:44.560629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.560605    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.578457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.578434    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f3c580-openshift-default-dockercfg-ws8ll\""
Oct 30 14:23:44.592006 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.591977    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"]
Oct 30 14:23:44.667998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.667959    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-envoy\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.668159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.668001    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-credential-socket\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.668159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.668092    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-podinfo\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.668159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.668126    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-certs\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.668316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.668291    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-socket\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.668502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.668484    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-data\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.668564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.668551    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-token\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.668623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.668606    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83aae0b-4976-4a91-937c-c1e3dda7f151-istiod-ca-cert\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.668692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.668675    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v88zt\" (UniqueName: \"kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-kube-api-access-v88zt\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.780699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.780599    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-envoy\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.782190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.782161    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-credential-socket\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.782402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.782382    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-podinfo\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.782531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.782517    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-certs\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.786628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.782751    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-socket\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.804901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.804862    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-data\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.807640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.807616    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-token\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.808185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.808166    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83aae0b-4976-4a91-937c-c1e3dda7f151-istiod-ca-cert\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.813336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.813317    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v88zt\" (UniqueName: \"kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-kube-api-access-v88zt\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.840266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.786330    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-certs\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.840266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.786952    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-socket\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.840266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.808954    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83aae0b-4976-4a91-937c-c1e3dda7f151-istiod-ca-cert\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.840266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.787202    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-credential-socket\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.840266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.805355    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-data\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.843449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.843420    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-podinfo\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.843879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.843857    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-envoy\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.865892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.865861    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-token\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:44.908567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:44.890114    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v88zt\" (UniqueName: \"kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-kube-api-access-v88zt\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:45.175915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:45.175884    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:45.494515 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:23:45.494488    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83aae0b_4976_4a91_937c_c1e3dda7f151.slice/crio-7703fbb71d345a426fc2ab0c36297ccaf3927a9dc2603ed2362e8b298a9fae24 WatchSource:0}: Error finding container 7703fbb71d345a426fc2ab0c36297ccaf3927a9dc2603ed2362e8b298a9fae24: Status 404 returned error can't find the container with id 7703fbb71d345a426fc2ab0c36297ccaf3927a9dc2603ed2362e8b298a9fae24
Oct 30 14:23:45.496636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:45.496612    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"]
Oct 30 14:23:45.497531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:45.497498    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:45.497618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:45.497581    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:45.497618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:45.497611    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:46.032927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.032884    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:46.164372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.162142    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:46.250175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.250138    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"]
Oct 30 14:23:46.254552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.254520    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.256670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.256582    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"]
Oct 30 14:23:46.272881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.272853    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-409364-openshift-default-dockercfg-r24w6\""
Oct 30 14:23:46.364816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.364777    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-credential-socket\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.364982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.364818    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-token\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.364982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.364853    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-envoy\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.365078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.365021    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-socket\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.365164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.365148    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88zwb\" (UniqueName: \"kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-kube-api-access-88zwb\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.365204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.365186    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-podinfo\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.365248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.365217    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-data\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.365347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.365331    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-certs\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.365384 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.365355    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istiod-ca-cert\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466081    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-socket\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466177    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88zwb\" (UniqueName: \"kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-kube-api-access-88zwb\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466206    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-podinfo\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466236    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-data\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466291    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-certs\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466312    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istiod-ca-cert\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466335    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-credential-socket\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466359    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-token\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466392    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-envoy\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466421    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-socket\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466651    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-data\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466805    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-credential-socket\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.466914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.466857    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-certs\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.467077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.467058    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istiod-ca-cert\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.468661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.468640    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-envoy\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.468796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.468779    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-podinfo\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.491789 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.491760    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" event={"ID":"a83aae0b-4976-4a91-937c-c1e3dda7f151","Type":"ContainerStarted","Data":"14b2a068f659197422cca734f2d5a9cf8ae07da516bad24d189aabe2f4be8445"}
Oct 30 14:23:46.491916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.491817    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" event={"ID":"a83aae0b-4976-4a91-937c-c1e3dda7f151","Type":"ContainerStarted","Data":"7703fbb71d345a426fc2ab0c36297ccaf3927a9dc2603ed2362e8b298a9fae24"}
Oct 30 14:23:46.493392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.493371    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-token\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.493511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.493489    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88zwb\" (UniqueName: \"kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-kube-api-access-88zwb\") pod \"e2e-inf-a-409364-openshift-default-868c776847-ntjrp\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") " pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.503363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.503341    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:23:46.544849 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.544800    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" podStartSLOduration=2.544787597 podStartE2EDuration="2.544787597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:46.541916864 +0000 UTC m=+2091.573815803" watchObservedRunningTime="2026-10-30 14:23:46.544787597 +0000 UTC m=+2091.576686536"
Oct 30 14:23:46.572644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.572620    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:46.900761 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:23:46.900732    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467f07a9_7aa2_4f27_9d2e_11106f3deb44.slice/crio-ea7cb4fb26e35d9d9160b67f72b3596cbb9be9a2262c4614d385013d8ba3e3c9 WatchSource:0}: Error finding container ea7cb4fb26e35d9d9160b67f72b3596cbb9be9a2262c4614d385013d8ba3e3c9: Status 404 returned error can't find the container with id ea7cb4fb26e35d9d9160b67f72b3596cbb9be9a2262c4614d385013d8ba3e3c9
Oct 30 14:23:46.903342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.903315    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"]
Oct 30 14:23:46.903985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.903949    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:46.904076 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.904028    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:46.904136 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:46.904075    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:47.176767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:47.176684    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:47.183398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:47.183373    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:47.501980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:47.501889    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" event={"ID":"467f07a9-7aa2-4f27-9d2e-11106f3deb44","Type":"ContainerStarted","Data":"487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312"}
Oct 30 14:23:47.501980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:47.501950    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" event={"ID":"467f07a9-7aa2-4f27-9d2e-11106f3deb44","Type":"ContainerStarted","Data":"ea7cb4fb26e35d9d9160b67f72b3596cbb9be9a2262c4614d385013d8ba3e3c9"}
Oct 30 14:23:47.502413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:47.502391    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:47.503231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:47.503213    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:23:47.554552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:47.554502    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" podStartSLOduration=2.55448894 podStartE2EDuration="2.55448894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:47.551568698 +0000 UTC m=+2092.583467683" watchObservedRunningTime="2026-10-30 14:23:47.55448894 +0000 UTC m=+2092.586387879"
Oct 30 14:23:47.573176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:47.573142    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:47.578754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:47.578735    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:48.506093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.506028    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:23:48.510843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.510815    4530 generic.go:356] "Generic (PLEG): container finished" podID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerID="445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26" exitCode=0
Oct 30 14:23:48.510963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.510867    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"
Oct 30 14:23:48.510963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.510897    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" event={"ID":"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a","Type":"ContainerDied","Data":"445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26"}
Oct 30 14:23:48.510963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.510950    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt" event={"ID":"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a","Type":"ContainerDied","Data":"58132235ac7ee15a738ce518edbcd7e12ccdd79970d7280caca660da6842f6f6"}
Oct 30 14:23:48.511091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.510971    4530 scope.go:122] "RemoveContainer" containerID="445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26"
Oct 30 14:23:48.511347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.511300    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:48.512252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.512238    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:23:48.521897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.521869    4530 scope.go:122] "RemoveContainer" containerID="9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22"
Oct 30 14:23:48.531792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.531776    4530 scope.go:122] "RemoveContainer" containerID="445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26"
Oct 30 14:23:48.532074 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:23:48.532056    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26\": container with ID starting with 445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26 not found: ID does not exist" containerID="445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26"
Oct 30 14:23:48.532138 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.532081    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26"} err="failed to get container status \"445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26\": rpc error: code = NotFound desc = could not find container \"445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26\": container with ID starting with 445346730c4bc118162b4e680bf81d62e1e5ab89f4b4bd16f4d2a60ebf7d1c26 not found: ID does not exist"
Oct 30 14:23:48.532138 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.532096    4530 scope.go:122] "RemoveContainer" containerID="9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22"
Oct 30 14:23:48.532346 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:23:48.532328    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22\": container with ID starting with 9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22 not found: ID does not exist" containerID="9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22"
Oct 30 14:23:48.532389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.532351    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22"} err="failed to get container status \"9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22\": rpc error: code = NotFound desc = could not find container \"9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22\": container with ID starting with 9e6847515faaa2d630124ead5c31a74b2ada54ee8fa093357c8abed614e27b22 not found: ID does not exist"
Oct 30 14:23:48.625883 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.625857    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-podinfo\") pod \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") "
Oct 30 14:23:48.626315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.625920    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-credential-socket\") pod \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") "
Oct 30 14:23:48.626315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.626022    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istiod-ca-cert\") pod \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") "
Oct 30 14:23:48.626315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.626127    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-envoy\") pod \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") "
Oct 30 14:23:48.626315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.626180    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-certs\") pod \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") "
Oct 30 14:23:48.626315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.626217    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-kube-api-access-xchg8\" (UniqueName: \"kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-kube-api-access-xchg8\") pod \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") "
Oct 30 14:23:48.626315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.626214    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-credential-socket" pod "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" (UID: "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:48.626315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.626257    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-token\" (UniqueName: \"kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-token\") pod \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") "
Oct 30 14:23:48.626761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.626350    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-socket\") pod \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") "
Oct 30 14:23:48.626761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.626382    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-data\") pod \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\" (UID: \"9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a\") "
Oct 30 14:23:48.626761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.626385    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istiod-ca-cert" pod "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" (UID: "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:23:48.626761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.626522    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-certs" pod "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" (UID: "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:48.626761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.626679    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-socket" pod "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" (UID: "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:48.627039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.626804    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-data" pod "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" (UID: "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:48.627257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.627234    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:48.627346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.627262    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:48.627346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.627277    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:48.627346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.627294    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:48.627346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.627306    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:48.628376 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.628345    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-podinfo" pod "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" (UID: "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:23:48.628670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.628653    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-token" pod "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" (UID: "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:23:48.628823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.628801    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-envoy" pod "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" (UID: "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:48.628927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.628911    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-kube-api-access-xchg8" pod "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" (UID: "9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a"). InnerVolumeSpecName "kube-api-access-xchg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:23:48.662873 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.662829    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:23:48.663005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.662911    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" podUID="23a039c6-aa25-40ac-ba46-8f0c59bef603" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:23:48.728701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.728669    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:48.728701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.728695    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xchg8\" (UniqueName: \"kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-kube-api-access-xchg8\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:48.728701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.728707    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:48.728943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.728716    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:48.866980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.866936    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"]
Oct 30 14:23:48.871623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:48.871598    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1c4b82-openshift-default-7f9f99f747-ppqbt"]
Oct 30 14:23:49.117197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.117123    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" path="/var/lib/kubelet/pods/9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a/volumes"
Oct 30 14:23:49.158010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.157978    4530 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:23:49.158159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.158032    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:23:49.774249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.774224    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:23:49.810886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.810866    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:23:49.844447 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.844417    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-credential-socket\") pod \"23a039c6-aa25-40ac-ba46-8f0c59bef603\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") "
Oct 30 14:23:49.844583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.844499    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-socket\") pod \"23a039c6-aa25-40ac-ba46-8f0c59bef603\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") "
Oct 30 14:23:49.844583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.844530    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-envoy\") pod \"23a039c6-aa25-40ac-ba46-8f0c59bef603\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") "
Oct 30 14:23:49.844713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.844650    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a039c6-aa25-40ac-ba46-8f0c59bef603-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a039c6-aa25-40ac-ba46-8f0c59bef603-istiod-ca-cert\") pod \"23a039c6-aa25-40ac-ba46-8f0c59bef603\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") "
Oct 30 14:23:49.844713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.844692    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-podinfo\") pod \"23a039c6-aa25-40ac-ba46-8f0c59bef603\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") "
Oct 30 14:23:49.844713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.844695    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-credential-socket" pod "23a039c6-aa25-40ac-ba46-8f0c59bef603" (UID: "23a039c6-aa25-40ac-ba46-8f0c59bef603"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:49.844891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.844733    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-kube-api-access-gqmvf\" (UniqueName: \"kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-kube-api-access-gqmvf\") pod \"23a039c6-aa25-40ac-ba46-8f0c59bef603\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") "
Oct 30 14:23:49.844891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.844880    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-certs\") pod \"23a039c6-aa25-40ac-ba46-8f0c59bef603\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") "
Oct 30 14:23:49.845008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.844934    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-data\") pod \"23a039c6-aa25-40ac-ba46-8f0c59bef603\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") "
Oct 30 14:23:49.845008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.844963    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-token\" (UniqueName: \"kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-token\") pod \"23a039c6-aa25-40ac-ba46-8f0c59bef603\" (UID: \"23a039c6-aa25-40ac-ba46-8f0c59bef603\") "
Oct 30 14:23:49.845107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.845044    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a039c6-aa25-40ac-ba46-8f0c59bef603-istiod-ca-cert" pod "23a039c6-aa25-40ac-ba46-8f0c59bef603" (UID: "23a039c6-aa25-40ac-ba46-8f0c59bef603"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:23:49.845173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.845123    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-socket" pod "23a039c6-aa25-40ac-ba46-8f0c59bef603" (UID: "23a039c6-aa25-40ac-ba46-8f0c59bef603"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:49.845278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.845254    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-certs" pod "23a039c6-aa25-40ac-ba46-8f0c59bef603" (UID: "23a039c6-aa25-40ac-ba46-8f0c59bef603"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:49.845602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.845579    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.845676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.845606    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.845676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.845619    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.845676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.845631    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a039c6-aa25-40ac-ba46-8f0c59bef603-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.845857 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.845838    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-data" pod "23a039c6-aa25-40ac-ba46-8f0c59bef603" (UID: "23a039c6-aa25-40ac-ba46-8f0c59bef603"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:49.847281 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.847257    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-envoy" pod "23a039c6-aa25-40ac-ba46-8f0c59bef603" (UID: "23a039c6-aa25-40ac-ba46-8f0c59bef603"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:49.847396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.847355    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-podinfo" pod "23a039c6-aa25-40ac-ba46-8f0c59bef603" (UID: "23a039c6-aa25-40ac-ba46-8f0c59bef603"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:23:49.848016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.847990    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-token" pod "23a039c6-aa25-40ac-ba46-8f0c59bef603" (UID: "23a039c6-aa25-40ac-ba46-8f0c59bef603"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:23:49.848115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.848052    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-kube-api-access-gqmvf" pod "23a039c6-aa25-40ac-ba46-8f0c59bef603" (UID: "23a039c6-aa25-40ac-ba46-8f0c59bef603"). InnerVolumeSpecName "kube-api-access-gqmvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:23:49.948067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.948031    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-credential-socket\") pod \"72a45514-45c1-441e-ad52-b48b6072eac8\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") "
Oct 30 14:23:49.948401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.948376    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-socket\") pod \"72a45514-45c1-441e-ad52-b48b6072eac8\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") "
Oct 30 14:23:49.948787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.948766    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a45514-45c1-441e-ad52-b48b6072eac8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a45514-45c1-441e-ad52-b48b6072eac8-istiod-ca-cert\") pod \"72a45514-45c1-441e-ad52-b48b6072eac8\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") "
Oct 30 14:23:49.949336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.949317    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-kube-api-access-rp8sp\" (UniqueName: \"kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-kube-api-access-rp8sp\") pod \"72a45514-45c1-441e-ad52-b48b6072eac8\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") "
Oct 30 14:23:49.949863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.949844    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-istio-token\" (UniqueName: \"kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-istio-token\") pod \"72a45514-45c1-441e-ad52-b48b6072eac8\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") "
Oct 30 14:23:49.950074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.950057    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-data\") pod \"72a45514-45c1-441e-ad52-b48b6072eac8\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") "
Oct 30 14:23:49.950617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.950599    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-certs\") pod \"72a45514-45c1-441e-ad52-b48b6072eac8\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") "
Oct 30 14:23:49.950993 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.950975    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72a45514-45c1-441e-ad52-b48b6072eac8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a45514-45c1-441e-ad52-b48b6072eac8-istio-podinfo\") pod \"72a45514-45c1-441e-ad52-b48b6072eac8\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") "
Oct 30 14:23:49.951427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.951407    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-envoy\") pod \"72a45514-45c1-441e-ad52-b48b6072eac8\" (UID: \"72a45514-45c1-441e-ad52-b48b6072eac8\") "
Oct 30 14:23:49.951891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.948447    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-credential-socket" pod "72a45514-45c1-441e-ad52-b48b6072eac8" (UID: "72a45514-45c1-441e-ad52-b48b6072eac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:49.951891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.949216    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a45514-45c1-441e-ad52-b48b6072eac8-istiod-ca-cert" pod "72a45514-45c1-441e-ad52-b48b6072eac8" (UID: "72a45514-45c1-441e-ad52-b48b6072eac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:23:49.951891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.950461    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-data" pod "72a45514-45c1-441e-ad52-b48b6072eac8" (UID: "72a45514-45c1-441e-ad52-b48b6072eac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:49.951891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.950914    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-certs" pod "72a45514-45c1-441e-ad52-b48b6072eac8" (UID: "72a45514-45c1-441e-ad52-b48b6072eac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:49.952510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.952486    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.952617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.952515    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.952680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.952617    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a45514-45c1-441e-ad52-b48b6072eac8-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.952680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.952635    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.952680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.952648    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.952680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.952661    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.952886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.952674    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a039c6-aa25-40ac-ba46-8f0c59bef603-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.952886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.952700    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmvf\" (UniqueName: \"kubernetes.io/projected/23a039c6-aa25-40ac-ba46-8f0c59bef603-kube-api-access-gqmvf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.952886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.952715    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:49.956021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.955995    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72a45514-45c1-441e-ad52-b48b6072eac8-istio-podinfo" pod "72a45514-45c1-441e-ad52-b48b6072eac8" (UID: "72a45514-45c1-441e-ad52-b48b6072eac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:23:49.956189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.956165    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-istio-token" pod "72a45514-45c1-441e-ad52-b48b6072eac8" (UID: "72a45514-45c1-441e-ad52-b48b6072eac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:23:49.956632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.956604    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-socket" pod "72a45514-45c1-441e-ad52-b48b6072eac8" (UID: "72a45514-45c1-441e-ad52-b48b6072eac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:49.956744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.956705    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-kube-api-access-rp8sp" pod "72a45514-45c1-441e-ad52-b48b6072eac8" (UID: "72a45514-45c1-441e-ad52-b48b6072eac8"). InnerVolumeSpecName "kube-api-access-rp8sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:23:49.957652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:49.957617    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-envoy" pod "72a45514-45c1-441e-ad52-b48b6072eac8" (UID: "72a45514-45c1-441e-ad52-b48b6072eac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:23:50.054476 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.054449    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a45514-45c1-441e-ad52-b48b6072eac8-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:50.054476 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.054474    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:50.054689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.054483    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a45514-45c1-441e-ad52-b48b6072eac8-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:50.054689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.054496    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp8sp\" (UniqueName: \"kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-kube-api-access-rp8sp\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:50.054689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.054510    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a45514-45c1-441e-ad52-b48b6072eac8-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:23:50.139957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.139931    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"]
Oct 30 14:23:50.140761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.140740    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerName="istio-proxy"
Oct 30 14:23:50.140761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.140761    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerName="istio-proxy"
Oct 30 14:23:50.140947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.140779    4530 cpu_manager.go:414] "containerMap: removing container" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy"
Oct 30 14:23:50.140947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.140787    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy"
Oct 30 14:23:50.140947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.140837    4530 cpu_manager.go:414] "containerMap: removing container" podUID="23a039c6-aa25-40ac-ba46-8f0c59bef603" containerName="istio-proxy"
Oct 30 14:23:50.140947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.140845    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a039c6-aa25-40ac-ba46-8f0c59bef603" containerName="istio-proxy"
Oct 30 14:23:50.141167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.141036    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerName="istio-proxy"
Oct 30 14:23:50.141167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.141052    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy"
Oct 30 14:23:50.141167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.141067    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy"
Oct 30 14:23:50.141167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.141078    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a039c6-aa25-40ac-ba46-8f0c59bef603" containerName="istio-proxy"
Oct 30 14:23:50.141167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.141092    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerName="istio-proxy"
Oct 30 14:23:50.145132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.145111    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.176153 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.174374    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"]
Oct 30 14:23:50.224432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.224396    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d15602-serving-cert\""
Oct 30 14:23:50.272239 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.272208    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-maas-api-tls\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.272394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.272246    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-metrics-certs\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.272440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.272403    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0da3cf-725c-4c81-9fff-0d0b202db949-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.272638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.272607    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm7t5\" (UniqueName: \"kubernetes.io/projected/aa0da3cf-725c-4c81-9fff-0d0b202db949-kube-api-access-sm7t5\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.374344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.374265    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0da3cf-725c-4c81-9fff-0d0b202db949-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.374519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.374388    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm7t5\" (UniqueName: \"kubernetes.io/projected/aa0da3cf-725c-4c81-9fff-0d0b202db949-kube-api-access-sm7t5\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.374519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.374428    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-maas-api-tls\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.374519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.374448    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-metrics-certs\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.374760 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:23:50.374587    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-serving-cert: secret "maas-api-e2e-inf-a-d15602-serving-cert" not found
Oct 30 14:23:50.374760 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:23:50.374657    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-maas-api-tls podName:aa0da3cf-725c-4c81-9fff-0d0b202db949 nodeName:}" failed. No retries permitted until 2026-10-30 14:23:50.874640078 +0000 UTC m=+2095.906538996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-maas-api-tls") pod "maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl" (UID: "aa0da3cf-725c-4c81-9fff-0d0b202db949") : secret "maas-api-e2e-inf-a-d15602-serving-cert" not found
Oct 30 14:23:50.375089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.375064    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0da3cf-725c-4c81-9fff-0d0b202db949-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.377055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.377034    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-metrics-certs\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.448567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.441352    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm7t5\" (UniqueName: \"kubernetes.io/projected/aa0da3cf-725c-4c81-9fff-0d0b202db949-kube-api-access-sm7t5\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.594577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.589715    4530 generic.go:356] "Generic (PLEG): container finished" podID="72a45514-45c1-441e-ad52-b48b6072eac8" containerID="76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a" exitCode=0
Oct 30 14:23:50.594577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.589903    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" event={"ID":"72a45514-45c1-441e-ad52-b48b6072eac8","Type":"ContainerDied","Data":"76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a"}
Oct 30 14:23:50.594577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.589961    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr" event={"ID":"72a45514-45c1-441e-ad52-b48b6072eac8","Type":"ContainerDied","Data":"029d9b11e040c9af0e4641418f798e9dc97cd9d9a76c91fb81a50861843632ab"}
Oct 30 14:23:50.594577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.589992    4530 scope.go:122] "RemoveContainer" containerID="76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a"
Oct 30 14:23:50.594577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.590142    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"
Oct 30 14:23:50.652589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.639478    4530 generic.go:356] "Generic (PLEG): container finished" podID="23a039c6-aa25-40ac-ba46-8f0c59bef603" containerID="db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69" exitCode=0
Oct 30 14:23:50.652589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.640765    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"
Oct 30 14:23:50.674565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.668181    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" event={"ID":"23a039c6-aa25-40ac-ba46-8f0c59bef603","Type":"ContainerDied","Data":"db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69"}
Oct 30 14:23:50.674565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.668251    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc" event={"ID":"23a039c6-aa25-40ac-ba46-8f0c59bef603","Type":"ContainerDied","Data":"67203a36c9cdb25dada4273212c861ef5077e4498fab92dc376da086e6061ccc"}
Oct 30 14:23:50.677752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.675698    4530 scope.go:122] "RemoveContainer" containerID="b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94"
Oct 30 14:23:50.692002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.691953    4530 scope.go:122] "RemoveContainer" containerID="76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a"
Oct 30 14:23:50.692271 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:23:50.692242    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a\": container with ID starting with 76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a not found: ID does not exist" containerID="76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a"
Oct 30 14:23:50.692341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.692278    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a"} err="failed to get container status \"76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a\": rpc error: code = NotFound desc = could not find container \"76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a\": container with ID starting with 76bc5198659b7caf88a0439647855ff0af9c004a7a8f0ce35f8799031dbe102a not found: ID does not exist"
Oct 30 14:23:50.692341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.692301    4530 scope.go:122] "RemoveContainer" containerID="b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94"
Oct 30 14:23:50.692615 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:23:50.692585    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94\": container with ID starting with b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94 not found: ID does not exist" containerID="b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94"
Oct 30 14:23:50.692664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.692624    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94"} err="failed to get container status \"b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94\": rpc error: code = NotFound desc = could not find container \"b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94\": container with ID starting with b2d493f1e8a709c47d90b7947edc20a34976288038b28e465bff2d26200dbe94 not found: ID does not exist"
Oct 30 14:23:50.692664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.692647    4530 scope.go:122] "RemoveContainer" containerID="db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69"
Oct 30 14:23:50.702573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.702533    4530 scope.go:122] "RemoveContainer" containerID="db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69"
Oct 30 14:23:50.702857 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:23:50.702835    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69\": container with ID starting with db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69 not found: ID does not exist" containerID="db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69"
Oct 30 14:23:50.702921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.702863    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69"} err="failed to get container status \"db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69\": rpc error: code = NotFound desc = could not find container \"db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69\": container with ID starting with db43a02466e7dfab620fcb9777dac1d42c5c4ecfe1082e1c78accd2477127e69 not found: ID does not exist"
Oct 30 14:23:50.736012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.735989    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"]
Oct 30 14:23:50.743901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.743879    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-tvnbr"]
Oct 30 14:23:50.785017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.784989    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"]
Oct 30 14:23:50.789308 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.789278    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-j46pc"]
Oct 30 14:23:50.893103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.893040    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-maas-api-tls\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:50.895344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:50.895325    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-maas-api-tls\") pod \"maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:51.073266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.073237    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:51.119024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.118995    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a039c6-aa25-40ac-ba46-8f0c59bef603" path="/var/lib/kubelet/pods/23a039c6-aa25-40ac-ba46-8f0c59bef603/volumes"
Oct 30 14:23:51.120100 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.119724    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" path="/var/lib/kubelet/pods/72a45514-45c1-441e-ad52-b48b6072eac8/volumes"
Oct 30 14:23:51.405948 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:23:51.405908    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0da3cf_725c_4c81_9fff_0d0b202db949.slice/crio-d7e167bdb4adcb4acc414269d41ce8e17d00e004121f667c25c653b92626be57 WatchSource:0}: Error finding container d7e167bdb4adcb4acc414269d41ce8e17d00e004121f667c25c653b92626be57: Status 404 returned error can't find the container with id d7e167bdb4adcb4acc414269d41ce8e17d00e004121f667c25c653b92626be57
Oct 30 14:23:51.410368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.410342    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"]
Oct 30 14:23:51.517696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.517661    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"]
Oct 30 14:23:51.518346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.518331    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerName="istio-proxy"
Oct 30 14:23:51.518403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.518347    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8c9ae2-bbe3-4d19-8360-3728e5ffa25a" containerName="istio-proxy"
Oct 30 14:23:51.518403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.518360    4530 cpu_manager.go:414] "containerMap: removing container" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy"
Oct 30 14:23:51.518403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.518365    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a45514-45c1-441e-ad52-b48b6072eac8" containerName="istio-proxy"
Oct 30 14:23:51.523640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.523616    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.559348 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.559312    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"]
Oct 30 14:23:51.745183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.745094    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.746854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.746817    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.747085 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.747069    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.747336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.747318    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.747482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.747468    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.747654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.747588    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.747893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.747873    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj6ph\" (UniqueName: \"kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-kube-api-access-pj6ph\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.751562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.748063    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.751773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.751750    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.863257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.863217    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.863797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.863289    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.863797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.863312    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.863797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.863395    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj6ph\" (UniqueName: \"kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-kube-api-access-pj6ph\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.863797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.863468    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.863797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.863516    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.863797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.863594    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.863797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.863629    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.863797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.863694    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.864979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.864947    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.866677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.866457    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.866798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.866777    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.867197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.867121    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.867425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.867370    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.882787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.882760    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.885823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.885797    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.889624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.889594    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl" event={"ID":"aa0da3cf-725c-4c81-9fff-0d0b202db949","Type":"ContainerStarted","Data":"d7e167bdb4adcb4acc414269d41ce8e17d00e004121f667c25c653b92626be57"}
Oct 30 14:23:51.890666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.890642    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj6ph\" (UniqueName: \"kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-kube-api-access-pj6ph\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:51.890959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:51.890938    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:52.142044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:52.141768    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:52.474237 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:23:52.474207    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8605fd0_853e_4cb6_9906_ceb6a07f8430.slice/crio-ee8e0a750b27be66bb643f04ad0581c8ea9703ea20d4ccf5bcdbd2dc7bf33281 WatchSource:0}: Error finding container ee8e0a750b27be66bb643f04ad0581c8ea9703ea20d4ccf5bcdbd2dc7bf33281: Status 404 returned error can't find the container with id ee8e0a750b27be66bb643f04ad0581c8ea9703ea20d4ccf5bcdbd2dc7bf33281
Oct 30 14:23:52.476492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:52.476465    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"]
Oct 30 14:23:52.477344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:52.477310    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:52.477404 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:52.477376    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:52.477448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:52.477407    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:23:52.953499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:52.953451    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl" event={"ID":"aa0da3cf-725c-4c81-9fff-0d0b202db949","Type":"ContainerStarted","Data":"6b080151439fad8352a933904c96831e11716e57fbf3098213c9eff02fb62f5c"}
Oct 30 14:23:52.954699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:52.954673    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:23:52.956998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:52.956963    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" event={"ID":"d8605fd0-853e-4cb6-9906-ceb6a07f8430","Type":"ContainerStarted","Data":"eea5cfc436f8fc748ac09214438dbae2e738ce3aee9f5a08be6c35dca8226025"}
Oct 30 14:23:52.957105 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:52.957013    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" event={"ID":"d8605fd0-853e-4cb6-9906-ceb6a07f8430","Type":"ContainerStarted","Data":"ee8e0a750b27be66bb643f04ad0581c8ea9703ea20d4ccf5bcdbd2dc7bf33281"}
Oct 30 14:23:53.004793 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:53.004747    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl" podStartSLOduration=2.50871441 podStartE2EDuration="3.004733662s" totalImagesPullingTime="496.019252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:53.003429107 +0000 UTC m=+2098.035328046" watchObservedRunningTime="2026-10-30 14:23:53.004733662 +0000 UTC m=+2098.036632601"
Oct 30 14:23:53.054319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:53.054276    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" podStartSLOduration=2.054263573 podStartE2EDuration="2.054263573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:53.053293327 +0000 UTC m=+2098.085192267" watchObservedRunningTime="2026-10-30 14:23:53.054263573 +0000 UTC m=+2098.086162511"
Oct 30 14:23:53.163935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:53.163900    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:23:53.197565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:53.181662    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 14:23:53.197565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:53.181769    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 14:23:54.142763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:54.142727    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 14:23:54.143206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:54.142797    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 14:23:55.142703 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:55.142665    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 14:23:55.142971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:55.142728    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 14:23:55.769630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:55.769603    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:23:55.769873 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:55.769858    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:23:55.774561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:55.774521    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:23:55.774756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:55.774739    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:23:55.775466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:55.775444    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:23:55.776125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:55.775655    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:23:56.143249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:56.143212    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 14:23:56.143698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:56.143278    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 14:23:57.142629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.142593    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 14:23:57.142809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.142654    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 14:23:57.625717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.625679    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"]
Oct 30 14:23:57.632069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.632041    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:57.653475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.652221    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"]
Oct 30 14:23:57.660588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.657435    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-34bcc5-serving-cert\""
Oct 30 14:23:57.717601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.717106    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-metrics-certs\") pod \"maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:57.718148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.718121    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxqdt\" (UniqueName: \"kubernetes.io/projected/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-kube-api-access-zxqdt\") pod \"maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:57.718307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.718234    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:57.718307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.718269    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:57.819415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.819387    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxqdt\" (UniqueName: \"kubernetes.io/projected/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-kube-api-access-zxqdt\") pod \"maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:57.819731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.819715    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:57.819835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.819821    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:57.819976 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.819966    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-metrics-certs\") pod \"maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:57.820797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.820767    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:57.822725 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.822703    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-metrics-certs\") pod \"maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:57.823891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.823865    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:57.846455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:57.846431    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxqdt\" (UniqueName: \"kubernetes.io/projected/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-kube-api-access-zxqdt\") pod \"maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:58.005103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:58.004933    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:58.143526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:58.143157    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 14:23:58.143526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:58.143218    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 14:23:58.376242 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:23:58.376215    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33b2b99_5155_44d9_b24f_e0de33e3e2fe.slice/crio-e18027f11cf815a30e0374ee5def699a689c57ee89b7ab9b960d57643ca9c9c4 WatchSource:0}: Error finding container e18027f11cf815a30e0374ee5def699a689c57ee89b7ab9b960d57643ca9c9c4: Status 404 returned error can't find the container with id e18027f11cf815a30e0374ee5def699a689c57ee89b7ab9b960d57643ca9c9c4
Oct 30 14:23:58.379211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:58.378683    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"]
Oct 30 14:23:59.054908 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:59.054867    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p" event={"ID":"d33b2b99-5155-44d9-b24f-e0de33e3e2fe","Type":"ContainerStarted","Data":"1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc"}
Oct 30 14:23:59.055318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:59.054936    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:23:59.055318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:59.054956    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p" event={"ID":"d33b2b99-5155-44d9-b24f-e0de33e3e2fe","Type":"ContainerStarted","Data":"e18027f11cf815a30e0374ee5def699a689c57ee89b7ab9b960d57643ca9c9c4"}
Oct 30 14:23:59.111461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:59.111400    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p" podStartSLOduration=1.584285488 podStartE2EDuration="2.111381899s" totalImagesPullingTime="527.096411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:59.110852133 +0000 UTC m=+2104.142751075" watchObservedRunningTime="2026-10-30 14:23:59.111381899 +0000 UTC m=+2104.143280841"
Oct 30 14:23:59.148565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:59.144951    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 14:23:59.148565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:59.145012    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 14:23:59.986130 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:23:59.986101    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:24:00.147621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:00.147584    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:24:00.148376 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:00.148354    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:24:00.149209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:00.149184    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:24:02.203424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:02.203387    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:24:05.076520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:05.076490    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:24:06.598157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.598118    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"]
Oct 30 14:24:06.604403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.604382    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.619666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.619629    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.619820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.619691    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.619820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.619724    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.620263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.620237    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.620618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.620593    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8bcc\" (UniqueName: \"kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-kube-api-access-w8bcc\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.620879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.620844    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.621018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.620911    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.621018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.620938    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.621018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.620961    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.626161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.626137    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"]
Oct 30 14:24:06.724923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.724867    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8bcc\" (UniqueName: \"kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-kube-api-access-w8bcc\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.725097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.725014    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.725097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.725058    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.725193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.725087    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.725193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.725182    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.725322 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.725308    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.725391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.725338    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.725391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.725365    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.725515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.725422    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.725515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.725488    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.725680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.725575    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.725955 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.725931    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.726169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.726146    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.726302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.726283    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.727526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.727506    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.728173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.728156    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.751934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.751903    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.752601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.752576    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8bcc\" (UniqueName: \"kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-kube-api-access-w8bcc\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:06.924318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:06.924229    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:07.255721 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:24:07.255686    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6a1bff_3d6b_4fbb_85b4_3f61ad374dc0.slice/crio-a3bfe72efcdd013046719ecef323d735a455788aba33f1dfeb9e0d8d61f580e6 WatchSource:0}: Error finding container a3bfe72efcdd013046719ecef323d735a455788aba33f1dfeb9e0d8d61f580e6: Status 404 returned error can't find the container with id a3bfe72efcdd013046719ecef323d735a455788aba33f1dfeb9e0d8d61f580e6
Oct 30 14:24:07.258658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.258610    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:07.258781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.258710    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:07.258781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.258761    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:07.261489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.261467    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"]
Oct 30 14:24:07.291274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.291248    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"]
Oct 30 14:24:07.291473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.291447    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29" podUID="4f414f83-9288-4b7d-9e7b-1170aa2beab0" containerName="maas-api" containerID="cri-o://93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1" gracePeriod=30
Oct 30 14:24:07.692621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.686197    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:24:07.778060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.778010    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-maas-api-tls\") pod \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") "
Oct 30 14:24:07.778326 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.778278    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f414f83-9288-4b7d-9e7b-1170aa2beab0-kube-api-access-vwrrq\" (UniqueName: \"kubernetes.io/projected/4f414f83-9288-4b7d-9e7b-1170aa2beab0-kube-api-access-vwrrq\") pod \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") "
Oct 30 14:24:07.778432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.778329    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-metrics-certs\") pod \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") "
Oct 30 14:24:07.778506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.778434    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f414f83-9288-4b7d-9e7b-1170aa2beab0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f414f83-9288-4b7d-9e7b-1170aa2beab0-openshift-service-ca\") pod \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\" (UID: \"4f414f83-9288-4b7d-9e7b-1170aa2beab0\") "
Oct 30 14:24:07.822562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.809741    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f414f83-9288-4b7d-9e7b-1170aa2beab0-openshift-service-ca" pod "4f414f83-9288-4b7d-9e7b-1170aa2beab0" (UID: "4f414f83-9288-4b7d-9e7b-1170aa2beab0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:24:07.855571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.855078    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f414f83-9288-4b7d-9e7b-1170aa2beab0-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:07.859845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.859793    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-maas-api-tls" pod "4f414f83-9288-4b7d-9e7b-1170aa2beab0" (UID: "4f414f83-9288-4b7d-9e7b-1170aa2beab0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:24:07.860480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.860444    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-metrics-certs" pod "4f414f83-9288-4b7d-9e7b-1170aa2beab0" (UID: "4f414f83-9288-4b7d-9e7b-1170aa2beab0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:24:07.942623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.940735    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f414f83-9288-4b7d-9e7b-1170aa2beab0-kube-api-access-vwrrq" pod "4f414f83-9288-4b7d-9e7b-1170aa2beab0" (UID: "4f414f83-9288-4b7d-9e7b-1170aa2beab0"). InnerVolumeSpecName "kube-api-access-vwrrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:24:07.979579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.963869    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:07.986574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.979985    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwrrq\" (UniqueName: \"kubernetes.io/projected/4f414f83-9288-4b7d-9e7b-1170aa2beab0-kube-api-access-vwrrq\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:07.986929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:07.986885    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f414f83-9288-4b7d-9e7b-1170aa2beab0-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:08.162277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.155004    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" event={"ID":"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0","Type":"ContainerStarted","Data":"10e6e3484a2613c41c0067ddf930c38793b8c945460f7421b21b79f3e3fffe97"}
Oct 30 14:24:08.162277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.155075    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" event={"ID":"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0","Type":"ContainerStarted","Data":"a3bfe72efcdd013046719ecef323d735a455788aba33f1dfeb9e0d8d61f580e6"}
Oct 30 14:24:08.162277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.157927    4530 generic.go:356] "Generic (PLEG): container finished" podID="4f414f83-9288-4b7d-9e7b-1170aa2beab0" containerID="93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1" exitCode=0
Oct 30 14:24:08.162277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.158107    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29" event={"ID":"4f414f83-9288-4b7d-9e7b-1170aa2beab0","Type":"ContainerDied","Data":"93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1"}
Oct 30 14:24:08.162277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.158161    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29" event={"ID":"4f414f83-9288-4b7d-9e7b-1170aa2beab0","Type":"ContainerDied","Data":"83945db42ecb3b6164edbe20208a7bb1a895bcf912950e8684d0af5be5bbffab"}
Oct 30 14:24:08.162277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.158195    4530 scope.go:122] "RemoveContainer" containerID="93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1"
Oct 30 14:24:08.162277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.158521    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"
Oct 30 14:24:08.212371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.212342    4530 scope.go:122] "RemoveContainer" containerID="93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1"
Oct 30 14:24:08.215388 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:24:08.215356    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1\": container with ID starting with 93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1 not found: ID does not exist" containerID="93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1"
Oct 30 14:24:08.215584 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.215527    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1"} err="failed to get container status \"93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1\": rpc error: code = NotFound desc = could not find container \"93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1\": container with ID starting with 93b53b7bb6c23741b3c0cdfedd920b64f480a4d98ec89b997c919bd4e60b41e1 not found: ID does not exist"
Oct 30 14:24:08.219293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.219250    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" podStartSLOduration=2.2192375970000002 podStartE2EDuration="2.219237597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:24:08.212634403 +0000 UTC m=+2113.244533344" watchObservedRunningTime="2026-10-30 14:24:08.219237597 +0000 UTC m=+2113.251136535"
Oct 30 14:24:08.277188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.277153    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"]
Oct 30 14:24:08.282530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.282502    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4bf33a-6bdd8855d9-fjq29"]
Oct 30 14:24:08.924593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.924560    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:08.926140 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.926114    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 14:24:08.926279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:08.926170    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" podUID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 14:24:09.118254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:09.118219    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f414f83-9288-4b7d-9e7b-1170aa2beab0" path="/var/lib/kubelet/pods/4f414f83-9288-4b7d-9e7b-1170aa2beab0/volumes"
Oct 30 14:24:09.924606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:09.924566    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 14:24:09.925026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:09.924624    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" podUID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 14:24:10.209067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.208977    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"]
Oct 30 14:24:10.213864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.211267    4530 cpu_manager.go:414] "containerMap: removing container" podUID="4f414f83-9288-4b7d-9e7b-1170aa2beab0" containerName="maas-api"
Oct 30 14:24:10.213864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.211307    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f414f83-9288-4b7d-9e7b-1170aa2beab0" containerName="maas-api"
Oct 30 14:24:10.213864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.211523    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f414f83-9288-4b7d-9e7b-1170aa2beab0" containerName="maas-api"
Oct 30 14:24:10.217623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.217598    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.220811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.220787    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"]
Oct 30 14:24:10.235973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.235938    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a9542c-serving-cert\""
Oct 30 14:24:10.331824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.331788    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.332112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.332086    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-metrics-certs\") pod \"maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.332199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.332179    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rmzv\" (UniqueName: \"kubernetes.io/projected/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-kube-api-access-8rmzv\") pod \"maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.332307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.332292    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-maas-api-tls\") pod \"maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.439562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.437570    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rmzv\" (UniqueName: \"kubernetes.io/projected/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-kube-api-access-8rmzv\") pod \"maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.439562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.437689    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-maas-api-tls\") pod \"maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.439562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.437767    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.456562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.448322    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.456562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.451869    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-metrics-certs\") pod \"maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.468802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.467258    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-metrics-certs\") pod \"maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.513566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.500166    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-maas-api-tls\") pod \"maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.534565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.530388    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rmzv\" (UniqueName: \"kubernetes.io/projected/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-kube-api-access-8rmzv\") pod \"maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.546182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.544656    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:10.907329 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:24:10.907281    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc93e64_3e9d_4e01_a0a8_27488bc4d312.slice/crio-94bd8179b50fe3b1ddafdedcd459ae4524c7c7d87c92cc5936f8fd4b4a364595 WatchSource:0}: Error finding container 94bd8179b50fe3b1ddafdedcd459ae4524c7c7d87c92cc5936f8fd4b4a364595: Status 404 returned error can't find the container with id 94bd8179b50fe3b1ddafdedcd459ae4524c7c7d87c92cc5936f8fd4b4a364595
Oct 30 14:24:10.909119 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.909093    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"]
Oct 30 14:24:10.925395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.925368    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 14:24:10.925744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:10.925408    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" podUID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 14:24:11.208029 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:11.207922    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n" event={"ID":"1bc93e64-3e9d-4e01-a0a8-27488bc4d312","Type":"ContainerStarted","Data":"94bd8179b50fe3b1ddafdedcd459ae4524c7c7d87c92cc5936f8fd4b4a364595"}
Oct 30 14:24:11.369764 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:24:11.369723    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:24:11.924650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:11.924616    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 14:24:11.924826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:11.924680    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" podUID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 14:24:12.218111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:12.218019    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n" event={"ID":"1bc93e64-3e9d-4e01-a0a8-27488bc4d312","Type":"ContainerStarted","Data":"b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477"}
Oct 30 14:24:12.218111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:12.218084    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:12.267887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:12.267830    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n" podStartSLOduration=1.7572455009999999 podStartE2EDuration="2.267816982s" totalImagesPullingTime="510.571481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:24:12.265437889 +0000 UTC m=+2117.297336829" watchObservedRunningTime="2026-10-30 14:24:12.267816982 +0000 UTC m=+2117.299715920"
Oct 30 14:24:12.924577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:12.924528    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 14:24:12.924765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:12.924598    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" podUID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 14:24:13.924687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:13.924643    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 14:24:13.925127 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:13.924709    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" podUID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 14:24:14.930369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:14.930333    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:14.930814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:14.930799    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:14.931576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:14.931522    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:24:18.232161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:18.232131    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:24:21.438756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:21.438725    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"]
Oct 30 14:24:21.439208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:21.438915    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" containerID="cri-o://11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045" gracePeriod=30
Oct 30 14:24:21.453047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:21.453009    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"]
Oct 30 14:24:21.453318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:21.453262    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy" containerID="cri-o://7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524" gracePeriod=30
Oct 30 14:24:21.458813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:21.458787    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"]
Oct 30 14:24:21.459000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:21.458981    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy" containerID="cri-o://7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67" gracePeriod=30
Oct 30 14:24:21.593155 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:21.593114    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-swl4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:24:21.593318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:21.593175    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:24:26.352764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.352730    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"]
Oct 30 14:24:26.354391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.354341    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy" containerID="cri-o://0422235b73ff130d48d4f305cf874f7ecacceb8e163252898771972b34be32aa" gracePeriod=30
Oct 30 14:24:26.867133 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.867073    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:24:26.876299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.876276    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:24:26.925927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.925893    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-podinfo\") pod \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.925945    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-kube-api-access-vcb8t\" (UniqueName: \"kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-kube-api-access-vcb8t\") pod \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.925992    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-data\") pod \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926054    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istiod-ca-cert\") pod \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926139    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-envoy\") pod \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926177    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-credential-socket\") pod \"7188a495-aaac-4e93-bbe4-0730b1190562\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926228    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-socket\") pod \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926259    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-istio-token\" (UniqueName: \"kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-istio-token\") pod \"7188a495-aaac-4e93-bbe4-0730b1190562\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926314    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-token\" (UniqueName: \"kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-token\") pod \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926343    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7188a495-aaac-4e93-bbe4-0730b1190562-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7188a495-aaac-4e93-bbe4-0730b1190562-istio-podinfo\") pod \"7188a495-aaac-4e93-bbe4-0730b1190562\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926389    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-envoy\") pod \"7188a495-aaac-4e93-bbe4-0730b1190562\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926471    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-kube-api-access-6fnrt\" (UniqueName: \"kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-kube-api-access-6fnrt\") pod \"7188a495-aaac-4e93-bbe4-0730b1190562\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926517    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-credential-socket\") pod \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926647    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-data\") pod \"7188a495-aaac-4e93-bbe4-0730b1190562\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926733    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7188a495-aaac-4e93-bbe4-0730b1190562-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7188a495-aaac-4e93-bbe4-0730b1190562-istiod-ca-cert\") pod \"7188a495-aaac-4e93-bbe4-0730b1190562\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926765    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-socket\") pod \"7188a495-aaac-4e93-bbe4-0730b1190562\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926809    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-certs\") pod \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\" (UID: \"0f237ea7-c1f7-46ba-ad61-a5871dbf3049\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.926853    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-certs\") pod \"7188a495-aaac-4e93-bbe4-0730b1190562\" (UID: \"7188a495-aaac-4e93-bbe4-0730b1190562\") "
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.927891    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-certs" pod "7188a495-aaac-4e93-bbe4-0730b1190562" (UID: "7188a495-aaac-4e93-bbe4-0730b1190562"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.929765    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-podinfo" pod "0f237ea7-c1f7-46ba-ad61-a5871dbf3049" (UID: "0f237ea7-c1f7-46ba-ad61-a5871dbf3049"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.930057    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-data" pod "7188a495-aaac-4e93-bbe4-0730b1190562" (UID: "7188a495-aaac-4e93-bbe4-0730b1190562"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.931063    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7188a495-aaac-4e93-bbe4-0730b1190562-istio-podinfo" pod "7188a495-aaac-4e93-bbe4-0730b1190562" (UID: "7188a495-aaac-4e93-bbe4-0730b1190562"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.931399    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7188a495-aaac-4e93-bbe4-0730b1190562-istiod-ca-cert" pod "7188a495-aaac-4e93-bbe4-0730b1190562" (UID: "7188a495-aaac-4e93-bbe4-0730b1190562"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.931698    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-socket" pod "7188a495-aaac-4e93-bbe4-0730b1190562" (UID: "7188a495-aaac-4e93-bbe4-0730b1190562"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.931941    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-certs" pod "0f237ea7-c1f7-46ba-ad61-a5871dbf3049" (UID: "0f237ea7-c1f7-46ba-ad61-a5871dbf3049"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.932622    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-socket" pod "0f237ea7-c1f7-46ba-ad61-a5871dbf3049" (UID: "0f237ea7-c1f7-46ba-ad61-a5871dbf3049"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.932801    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-data" pod "0f237ea7-c1f7-46ba-ad61-a5871dbf3049" (UID: "0f237ea7-c1f7-46ba-ad61-a5871dbf3049"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.932939    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-envoy" pod "7188a495-aaac-4e93-bbe4-0730b1190562" (UID: "7188a495-aaac-4e93-bbe4-0730b1190562"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.933042    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istiod-ca-cert" pod "0f237ea7-c1f7-46ba-ad61-a5871dbf3049" (UID: "0f237ea7-c1f7-46ba-ad61-a5871dbf3049"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.935087    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-envoy" pod "0f237ea7-c1f7-46ba-ad61-a5871dbf3049" (UID: "0f237ea7-c1f7-46ba-ad61-a5871dbf3049"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.935385    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-credential-socket" pod "7188a495-aaac-4e93-bbe4-0730b1190562" (UID: "7188a495-aaac-4e93-bbe4-0730b1190562"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.935730    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-credential-socket" pod "0f237ea7-c1f7-46ba-ad61-a5871dbf3049" (UID: "0f237ea7-c1f7-46ba-ad61-a5871dbf3049"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.935855    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-token" pod "0f237ea7-c1f7-46ba-ad61-a5871dbf3049" (UID: "0f237ea7-c1f7-46ba-ad61-a5871dbf3049"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.936981    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-kube-api-access-6fnrt" pod "7188a495-aaac-4e93-bbe4-0730b1190562" (UID: "7188a495-aaac-4e93-bbe4-0730b1190562"). InnerVolumeSpecName "kube-api-access-6fnrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:24:26.939398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.937414    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-kube-api-access-vcb8t" pod "0f237ea7-c1f7-46ba-ad61-a5871dbf3049" (UID: "0f237ea7-c1f7-46ba-ad61-a5871dbf3049"). InnerVolumeSpecName "kube-api-access-vcb8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:24:26.941375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:26.939524    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-istio-token" pod "7188a495-aaac-4e93-bbe4-0730b1190562" (UID: "7188a495-aaac-4e93-bbe4-0730b1190562"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:24:27.029489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029456    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7188a495-aaac-4e93-bbe4-0730b1190562-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029493    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029510    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029524    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029553    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029567    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcb8t\" (UniqueName: \"kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-kube-api-access-vcb8t\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029582    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029595    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029608    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029621    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029634    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029647    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029659    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029670    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7188a495-aaac-4e93-bbe4-0730b1190562-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.029670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029683    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.030311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029697    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fnrt\" (UniqueName: \"kubernetes.io/projected/7188a495-aaac-4e93-bbe4-0730b1190562-kube-api-access-6fnrt\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.030311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029711    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f237ea7-c1f7-46ba-ad61-a5871dbf3049-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.030311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.029725    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7188a495-aaac-4e93-bbe4-0730b1190562-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.189101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.189076    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:24:27.231519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.231491    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-kube-api-access-wbtmf\" (UniqueName: \"kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-kube-api-access-wbtmf\") pod \"a8c719d3-00ae-46f7-8364-6af8392425b5\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") "
Oct 30 14:24:27.231689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.231534    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-socket\") pod \"a8c719d3-00ae-46f7-8364-6af8392425b5\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") "
Oct 30 14:24:27.231689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.231602    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c719d3-00ae-46f7-8364-6af8392425b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c719d3-00ae-46f7-8364-6af8392425b5-istiod-ca-cert\") pod \"a8c719d3-00ae-46f7-8364-6af8392425b5\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") "
Oct 30 14:24:27.231689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.231632    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-certs\") pod \"a8c719d3-00ae-46f7-8364-6af8392425b5\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") "
Oct 30 14:24:27.231689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.231662    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-envoy\") pod \"a8c719d3-00ae-46f7-8364-6af8392425b5\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") "
Oct 30 14:24:27.231689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.231695    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-podinfo\") pod \"a8c719d3-00ae-46f7-8364-6af8392425b5\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") "
Oct 30 14:24:27.231979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.231770    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-data\") pod \"a8c719d3-00ae-46f7-8364-6af8392425b5\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") "
Oct 30 14:24:27.231979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.231911    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-socket" pod "a8c719d3-00ae-46f7-8364-6af8392425b5" (UID: "a8c719d3-00ae-46f7-8364-6af8392425b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:27.231979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.231926    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-token\" (UniqueName: \"kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-token\") pod \"a8c719d3-00ae-46f7-8364-6af8392425b5\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") "
Oct 30 14:24:27.231979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.231929    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-certs" pod "a8c719d3-00ae-46f7-8364-6af8392425b5" (UID: "a8c719d3-00ae-46f7-8364-6af8392425b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:27.231979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.231965    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-credential-socket\") pod \"a8c719d3-00ae-46f7-8364-6af8392425b5\" (UID: \"a8c719d3-00ae-46f7-8364-6af8392425b5\") "
Oct 30 14:24:27.232256 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.232083    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c719d3-00ae-46f7-8364-6af8392425b5-istiod-ca-cert" pod "a8c719d3-00ae-46f7-8364-6af8392425b5" (UID: "a8c719d3-00ae-46f7-8364-6af8392425b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:24:27.232256 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.232176    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-data" pod "a8c719d3-00ae-46f7-8364-6af8392425b5" (UID: "a8c719d3-00ae-46f7-8364-6af8392425b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:27.232413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.232390    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-credential-socket" pod "a8c719d3-00ae-46f7-8364-6af8392425b5" (UID: "a8c719d3-00ae-46f7-8364-6af8392425b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:27.232697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.232674    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.232697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.232696    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.232854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.232709    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c719d3-00ae-46f7-8364-6af8392425b5-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.232854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.232722    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.232854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.232731    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.234039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.234015    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-podinfo" pod "a8c719d3-00ae-46f7-8364-6af8392425b5" (UID: "a8c719d3-00ae-46f7-8364-6af8392425b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:24:27.234124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.234044    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-token" pod "a8c719d3-00ae-46f7-8364-6af8392425b5" (UID: "a8c719d3-00ae-46f7-8364-6af8392425b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:24:27.234248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.234231    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-envoy" pod "a8c719d3-00ae-46f7-8364-6af8392425b5" (UID: "a8c719d3-00ae-46f7-8364-6af8392425b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:27.234289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.234263    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-kube-api-access-wbtmf" pod "a8c719d3-00ae-46f7-8364-6af8392425b5" (UID: "a8c719d3-00ae-46f7-8364-6af8392425b5"). InnerVolumeSpecName "kube-api-access-wbtmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:24:27.333957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.333926    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.333957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.333952    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbtmf\" (UniqueName: \"kubernetes.io/projected/a8c719d3-00ae-46f7-8364-6af8392425b5-kube-api-access-wbtmf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.333957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.333962    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.334173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.333970    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c719d3-00ae-46f7-8364-6af8392425b5-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:27.403365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.403280    4530 generic.go:356] "Generic (PLEG): container finished" podID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerID="7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67" exitCode=0
Oct 30 14:24:27.403365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.403329    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"
Oct 30 14:24:27.403882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.403367    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" event={"ID":"a8c719d3-00ae-46f7-8364-6af8392425b5","Type":"ContainerDied","Data":"7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67"}
Oct 30 14:24:27.403882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.403422    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk" event={"ID":"a8c719d3-00ae-46f7-8364-6af8392425b5","Type":"ContainerDied","Data":"e675691c1e79e4b8c9dbe8310f814c07203608eaa6b16c888537172dc1c35fa0"}
Oct 30 14:24:27.403882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.403445    4530 scope.go:122] "RemoveContainer" containerID="7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67"
Oct 30 14:24:27.405220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.405199    4530 generic.go:356] "Generic (PLEG): container finished" podID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerID="7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524" exitCode=0
Oct 30 14:24:27.405294 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.405241    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"
Oct 30 14:24:27.405337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.405306    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" event={"ID":"0f237ea7-c1f7-46ba-ad61-a5871dbf3049","Type":"ContainerDied","Data":"7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524"}
Oct 30 14:24:27.405403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.405348    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg" event={"ID":"0f237ea7-c1f7-46ba-ad61-a5871dbf3049","Type":"ContainerDied","Data":"442d8d2699b8733fb352910ac4c3306cee230ea05e73cdb400fec1e7e6f839a9"}
Oct 30 14:24:27.406953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.406932    4530 generic.go:356] "Generic (PLEG): container finished" podID="7188a495-aaac-4e93-bbe4-0730b1190562" containerID="11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045" exitCode=0
Oct 30 14:24:27.407075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.406989    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"
Oct 30 14:24:27.407075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.407027    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" event={"ID":"7188a495-aaac-4e93-bbe4-0730b1190562","Type":"ContainerDied","Data":"11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045"}
Oct 30 14:24:27.407075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.407072    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n" event={"ID":"7188a495-aaac-4e93-bbe4-0730b1190562","Type":"ContainerDied","Data":"8b5ae598309853892009ce9430fefc9a49d54f70191a1a82b7f26f0fd2c60990"}
Oct 30 14:24:27.416876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.416848    4530 scope.go:122] "RemoveContainer" containerID="7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67"
Oct 30 14:24:27.417120 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:24:27.417101    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67\": container with ID starting with 7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67 not found: ID does not exist" containerID="7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67"
Oct 30 14:24:27.417205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.417133    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67"} err="failed to get container status \"7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67\": rpc error: code = NotFound desc = could not find container \"7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67\": container with ID starting with 7551c2675f01008c5a192626545424496a19bbf8bd8672a212e703875ce8ed67 not found: ID does not exist"
Oct 30 14:24:27.417205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.417157    4530 scope.go:122] "RemoveContainer" containerID="7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524"
Oct 30 14:24:27.426919 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.426896    4530 scope.go:122] "RemoveContainer" containerID="7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524"
Oct 30 14:24:27.427186 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:24:27.427164    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524\": container with ID starting with 7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524 not found: ID does not exist" containerID="7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524"
Oct 30 14:24:27.427232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.427193    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524"} err="failed to get container status \"7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524\": rpc error: code = NotFound desc = could not find container \"7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524\": container with ID starting with 7e9f62b954670ecc9e436dd80a4113df66736d5eb1a5a0903d8e8d85f7bb2524 not found: ID does not exist"
Oct 30 14:24:27.427232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.427208    4530 scope.go:122] "RemoveContainer" containerID="11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045"
Oct 30 14:24:27.436741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.436722    4530 scope.go:122] "RemoveContainer" containerID="11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045"
Oct 30 14:24:27.437011 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:24:27.436991    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045\": container with ID starting with 11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045 not found: ID does not exist" containerID="11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045"
Oct 30 14:24:27.437092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.437017    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045"} err="failed to get container status \"11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045\": rpc error: code = NotFound desc = could not find container \"11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045\": container with ID starting with 11dce961176eeb63338f310ab850a445ecd174611c9fe97837f27e1fe2221045 not found: ID does not exist"
Oct 30 14:24:27.468997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.468978    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"]
Oct 30 14:24:27.473179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.473161    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-swl4n"]
Oct 30 14:24:27.516647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.516623    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"]
Oct 30 14:24:27.520791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.520771    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6gclk"]
Oct 30 14:24:27.563313 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.563289    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"]
Oct 30 14:24:27.568069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:27.568050    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7mtkg"]
Oct 30 14:24:29.116314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:29.116282    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" path="/var/lib/kubelet/pods/0f237ea7-c1f7-46ba-ad61-a5871dbf3049/volumes"
Oct 30 14:24:29.116890 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:29.116866    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" path="/var/lib/kubelet/pods/7188a495-aaac-4e93-bbe4-0730b1190562/volumes"
Oct 30 14:24:29.117373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:29.117358    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" path="/var/lib/kubelet/pods/a8c719d3-00ae-46f7-8364-6af8392425b5/volumes"
Oct 30 14:24:31.446571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.446509    4530 generic.go:356] "Generic (PLEG): container finished" podID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerID="0422235b73ff130d48d4f305cf874f7ecacceb8e163252898771972b34be32aa" exitCode=0
Oct 30 14:24:31.446979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.446590    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" event={"ID":"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472","Type":"ContainerDied","Data":"0422235b73ff130d48d4f305cf874f7ecacceb8e163252898771972b34be32aa"}
Oct 30 14:24:31.446979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.446652    4530 scope.go:122] "RemoveContainer" containerID="7a86d1bebd2b06b31b251cda9d9c22c43c4533959a6f0a34b5a6df2a866bbb43"
Oct 30 14:24:31.584372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.584347    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:24:31.717988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.717909    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-token\" (UniqueName: \"kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-token\") pod \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") "
Oct 30 14:24:31.717988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.717981    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-certs\") pod \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") "
Oct 30 14:24:31.718220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.718038    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-kube-api-access-9nfck\" (UniqueName: \"kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-kube-api-access-9nfck\") pod \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") "
Oct 30 14:24:31.718220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.718076    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-data\") pod \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") "
Oct 30 14:24:31.718220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.718120    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-podinfo\") pod \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") "
Oct 30 14:24:31.718220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.718201    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-envoy\") pod \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") "
Oct 30 14:24:31.718436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.718267    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-certs" pod "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" (UID: "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:31.718436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.718358    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-credential-socket\") pod \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") "
Oct 30 14:24:31.718436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.718374    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-data" pod "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" (UID: "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:31.718653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.718441    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istiod-ca-cert\") pod \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") "
Oct 30 14:24:31.718717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.718651    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-socket\") pod \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\" (UID: \"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472\") "
Oct 30 14:24:31.718717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.718659    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-credential-socket" pod "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" (UID: "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:31.718923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.718894    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istiod-ca-cert" pod "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" (UID: "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:24:31.719021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.718933    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-socket" pod "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" (UID: "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:31.719433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.719411    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:31.719472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.719443    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:31.719472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.719459    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:31.719559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.719474    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:31.719559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.719489    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:31.720364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.720335    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-kube-api-access-9nfck" pod "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" (UID: "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472"). InnerVolumeSpecName "kube-api-access-9nfck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:24:31.720466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.720450    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-envoy" pod "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" (UID: "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:24:31.720718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.720688    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-podinfo" pod "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" (UID: "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:24:31.720809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.720718    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-token" pod "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" (UID: "a2d7564b-c0c0-4031-b7e9-d77bfe2b7472"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:24:31.820531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.820501    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:31.820531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.820528    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nfck\" (UniqueName: \"kubernetes.io/projected/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-kube-api-access-9nfck\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:31.820531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.820551    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:31.820764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:31.820565    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:24:32.372710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.372678    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"]
Oct 30 14:24:32.373368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373344    4530 cpu_manager.go:414] "containerMap: removing container" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy"
Oct 30 14:24:32.373368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373361    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy"
Oct 30 14:24:32.373568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373377    4530 cpu_manager.go:414] "containerMap: removing container" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy"
Oct 30 14:24:32.373568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373382    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy"
Oct 30 14:24:32.373568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373404    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy"
Oct 30 14:24:32.373568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373410    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy"
Oct 30 14:24:32.373568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373426    4530 cpu_manager.go:414] "containerMap: removing container" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy"
Oct 30 14:24:32.373568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373430    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy"
Oct 30 14:24:32.373568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373442    4530 cpu_manager.go:414] "containerMap: removing container" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy"
Oct 30 14:24:32.373568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373449    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy"
Oct 30 14:24:32.373927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373610    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f237ea7-c1f7-46ba-ad61-a5871dbf3049" containerName="istio-proxy"
Oct 30 14:24:32.373927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373624    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c719d3-00ae-46f7-8364-6af8392425b5" containerName="istio-proxy"
Oct 30 14:24:32.373927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373633    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7188a495-aaac-4e93-bbe4-0730b1190562" containerName="istio-proxy"
Oct 30 14:24:32.373927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373642    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy"
Oct 30 14:24:32.373927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.373650    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" containerName="istio-proxy"
Oct 30 14:24:32.377646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.377624    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.397921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.397901    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"]
Oct 30 14:24:32.405252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.405217    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f3c580-serving-cert\""
Oct 30 14:24:32.427520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.427492    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.427705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.427684    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-metrics-certs\") pod \"maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.427791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.427727    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-maas-api-tls\") pod \"maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.427791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.427758    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwgm2\" (UniqueName: \"kubernetes.io/projected/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-kube-api-access-vwgm2\") pod \"maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.457117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.457096    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"
Oct 30 14:24:32.457498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.457126    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9" event={"ID":"a2d7564b-c0c0-4031-b7e9-d77bfe2b7472","Type":"ContainerDied","Data":"e5d050853d711df24bdeb140769ee1caa196ab0318f66b9e928b77ad2faca7c5"}
Oct 30 14:24:32.457498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.457184    4530 scope.go:122] "RemoveContainer" containerID="0422235b73ff130d48d4f305cf874f7ecacceb8e163252898771972b34be32aa"
Oct 30 14:24:32.522957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.522924    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"]
Oct 30 14:24:32.528887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.528853    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.529040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.529020    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-metrics-certs\") pod \"maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.529166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.529062    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-maas-api-tls\") pod \"maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.529233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.529201    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwgm2\" (UniqueName: \"kubernetes.io/projected/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-kube-api-access-vwgm2\") pod \"maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.529683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.529659    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.530500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.530479    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4bf33a-openshift-default-b849dbcc5-ttvq9"]
Oct 30 14:24:32.531498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.531480    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-metrics-certs\") pod \"maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.531602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.531523    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-maas-api-tls\") pod \"maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.555680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.555658    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwgm2\" (UniqueName: \"kubernetes.io/projected/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-kube-api-access-vwgm2\") pod \"maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:32.693109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:32.693032    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:33.016266 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:24:33.004282    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e10d2d_7b12_4cad_bcb9_b9582142b2b8.slice/crio-f04deb4522d768fa9b4ba06254994219c1a3e2c2a3f6ae8bf811f74198356c84 WatchSource:0}: Error finding container f04deb4522d768fa9b4ba06254994219c1a3e2c2a3f6ae8bf811f74198356c84: Status 404 returned error can't find the container with id f04deb4522d768fa9b4ba06254994219c1a3e2c2a3f6ae8bf811f74198356c84
Oct 30 14:24:33.016266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:33.009028    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"]
Oct 30 14:24:33.115669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:33.115644    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d7564b-c0c0-4031-b7e9-d77bfe2b7472" path="/var/lib/kubelet/pods/a2d7564b-c0c0-4031-b7e9-d77bfe2b7472/volumes"
Oct 30 14:24:33.466249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:33.466221    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs" event={"ID":"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8","Type":"ContainerStarted","Data":"f04deb4522d768fa9b4ba06254994219c1a3e2c2a3f6ae8bf811f74198356c84"}
Oct 30 14:24:34.476616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:34.476573    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs" event={"ID":"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8","Type":"ContainerStarted","Data":"746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91"}
Oct 30 14:24:34.477020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:34.476669    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:34.526248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:34.526196    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs" podStartSLOduration=2.081234191 podStartE2EDuration="2.526180299s" totalImagesPullingTime="444.946108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:24:34.524111429 +0000 UTC m=+2139.556010367" watchObservedRunningTime="2026-10-30 14:24:34.526180299 +0000 UTC m=+2139.558079237"
Oct 30 14:24:36.585352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.585315    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"]
Oct 30 14:24:36.590007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.589990    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:36.608010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.607985    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"]
Oct 30 14:24:36.619609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.619509    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-409364-serving-cert\""
Oct 30 14:24:36.693566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.693499    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wd7x\" (UniqueName: \"kubernetes.io/projected/a381d859-fd85-4ec4-be51-3f6d6902cd01-kube-api-access-9wd7x\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:36.693719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.693649    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-maas-api-tls\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:36.693719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.693681    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-metrics-certs\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:36.693830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.693805    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a381d859-fd85-4ec4-be51-3f6d6902cd01-openshift-service-ca\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:36.803563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.799011    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a381d859-fd85-4ec4-be51-3f6d6902cd01-openshift-service-ca\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:36.803563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.799307    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wd7x\" (UniqueName: \"kubernetes.io/projected/a381d859-fd85-4ec4-be51-3f6d6902cd01-kube-api-access-9wd7x\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:36.803563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.799468    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-maas-api-tls\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:36.803563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.799512    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-metrics-certs\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:36.803563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.800653    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a381d859-fd85-4ec4-be51-3f6d6902cd01-openshift-service-ca\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:36.803563 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:24:36.800755    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-serving-cert: secret "maas-api-e2e-inf-a-409364-serving-cert" not found
Oct 30 14:24:36.803563 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:24:36.800813    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-maas-api-tls podName:a381d859-fd85-4ec4-be51-3f6d6902cd01 nodeName:}" failed. No retries permitted until 2026-10-30 14:24:37.300794822 +0000 UTC m=+2142.332693743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-maas-api-tls") pod "maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn" (UID: "a381d859-fd85-4ec4-be51-3f6d6902cd01") : secret "maas-api-e2e-inf-a-409364-serving-cert" not found
Oct 30 14:24:36.839610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.834171    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-metrics-certs\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:36.856357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:36.856318    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wd7x\" (UniqueName: \"kubernetes.io/projected/a381d859-fd85-4ec4-be51-3f6d6902cd01-kube-api-access-9wd7x\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:37.307120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:37.307076    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-maas-api-tls\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:37.309693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:37.309666    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-maas-api-tls\") pod \"maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:37.506391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:37.506358    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:37.946598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:37.943082    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"]
Oct 30 14:24:38.006968 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:24:38.006933    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda381d859_fd85_4ec4_be51_3f6d6902cd01.slice/crio-1b1636059fb49502e2c9a158117c4a607467cda4098cec1379630038c9a0e3af WatchSource:0}: Error finding container 1b1636059fb49502e2c9a158117c4a607467cda4098cec1379630038c9a0e3af: Status 404 returned error can't find the container with id 1b1636059fb49502e2c9a158117c4a607467cda4098cec1379630038c9a0e3af
Oct 30 14:24:38.565421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:38.565377    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn" event={"ID":"a381d859-fd85-4ec4-be51-3f6d6902cd01","Type":"ContainerStarted","Data":"1b1636059fb49502e2c9a158117c4a607467cda4098cec1379630038c9a0e3af"}
Oct 30 14:24:39.632566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:39.629063    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn" event={"ID":"a381d859-fd85-4ec4-be51-3f6d6902cd01","Type":"ContainerStarted","Data":"ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24"}
Oct 30 14:24:39.632566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:39.629659    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:39.693567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:39.691854    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn" podStartSLOduration=3.292701986 podStartE2EDuration="3.691836094s" totalImagesPullingTime="399.134108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:24:39.678065297 +0000 UTC m=+2144.709964238" watchObservedRunningTime="2026-10-30 14:24:39.691836094 +0000 UTC m=+2144.723735033"
Oct 30 14:24:40.522871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:40.522835    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:24:41.401801 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:24:41.401756    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:24:45.657661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:45.657630    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:24:46.650821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.650788    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"]
Oct 30 14:24:46.657951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.657922    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.678500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.678466    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"]
Oct 30 14:24:46.709566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.687785    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-93549e-openshift-default-dockercfg-r6j9x\""
Oct 30 14:24:46.713572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.710873    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-data\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.713853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.713825    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-credential-socket\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.713987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.713970    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-certs\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.714139 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.714125    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acdd5958-7f02-448b-8bcc-654433ec1cd2-istiod-ca-cert\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.714329 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.714313    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-envoy\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.714508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.714492    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-token\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.714787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.714760    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-socket\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.714860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.714835    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgczq\" (UniqueName: \"kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-kube-api-access-pgczq\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.714929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.714911    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-podinfo\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.875495    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgczq\" (UniqueName: \"kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-kube-api-access-pgczq\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.875595    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-podinfo\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.875667    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-data\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.875753    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-credential-socket\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.875795    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-certs\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.875866    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acdd5958-7f02-448b-8bcc-654433ec1cd2-istiod-ca-cert\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.875968    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-envoy\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.876060    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-token\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.876105    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-socket\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.876249    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-data\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.876512    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-socket\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.876789    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-credential-socket\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.877332    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acdd5958-7f02-448b-8bcc-654433ec1cd2-istiod-ca-cert\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.884572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.877783    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-certs\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:46.977610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:46.944272    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgczq\" (UniqueName: \"kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-kube-api-access-pgczq\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:47.075641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:47.072475    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-envoy\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:47.076336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:47.076301    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-podinfo\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:47.094519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:47.094480    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-token\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:47.335561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:47.333838    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:47.782811 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:24:47.782761    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacdd5958_7f02_448b_8bcc_654433ec1cd2.slice/crio-42bbba1d277d81eac910f560cc8cfd3efa42ddcc42f853d7f582cae9bfaa89cd WatchSource:0}: Error finding container 42bbba1d277d81eac910f560cc8cfd3efa42ddcc42f853d7f582cae9bfaa89cd: Status 404 returned error can't find the container with id 42bbba1d277d81eac910f560cc8cfd3efa42ddcc42f853d7f582cae9bfaa89cd
Oct 30 14:24:47.800516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:47.787838    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"]
Oct 30 14:24:47.800516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:47.788654    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:47.800516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:47.788766    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:47.800516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:47.788836    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:48.114181 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:48.113044    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" event={"ID":"acdd5958-7f02-448b-8bcc-654433ec1cd2","Type":"ContainerStarted","Data":"2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f"}
Oct 30 14:24:48.114181 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:48.113105    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" event={"ID":"acdd5958-7f02-448b-8bcc-654433ec1cd2","Type":"ContainerStarted","Data":"42bbba1d277d81eac910f560cc8cfd3efa42ddcc42f853d7f582cae9bfaa89cd"}
Oct 30 14:24:48.177731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:48.177678    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" podStartSLOduration=2.17766288 podStartE2EDuration="2.17766288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:24:48.173377103 +0000 UTC m=+2153.205276044" watchObservedRunningTime="2026-10-30 14:24:48.17766288 +0000 UTC m=+2153.209561818"
Oct 30 14:24:48.336679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:48.335008    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:48.337380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:48.337026    4530 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 14:24:48.337380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:48.337087    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 14:24:49.343752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:49.343720    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:50.134009 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:50.133970    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:50.135154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:50.135128    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:24:52.214237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:52.151452    4530 generic.go:356] "Generic (PLEG): container finished" podID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerID="3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62" exitCode=0
Oct 30 14:24:52.214237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:52.151521    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" event={"ID":"28a281b3-e4e7-47ff-83b7-c34d7866ac19","Type":"ContainerDied","Data":"3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62"}
Oct 30 14:24:52.214237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:52.152170    4530 scope.go:122] "RemoveContainer" containerID="3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62"
Oct 30 14:24:52.214237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:52.154054    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:52.214237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:52.154170    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:52.214237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:52.154220    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:53.160806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.160772    4530 generic.go:356] "Generic (PLEG): container finished" podID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerID="601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3" exitCode=0
Oct 30 14:24:53.161043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.160845    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" event={"ID":"3310a9c6-08cc-46ef-8a4a-27e960c28216","Type":"ContainerDied","Data":"601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3"}
Oct 30 14:24:53.161313 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.161291    4530 scope.go:122] "RemoveContainer" containerID="601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3"
Oct 30 14:24:53.162662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.162639    4530 generic.go:356] "Generic (PLEG): container finished" podID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerID="559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69" exitCode=0
Oct 30 14:24:53.162748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.162705    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" event={"ID":"89261fc0-4ce9-4d2a-b886-f730e9424523","Type":"ContainerDied","Data":"559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69"}
Oct 30 14:24:53.163074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.163054    4530 scope.go:122] "RemoveContainer" containerID="559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69"
Oct 30 14:24:53.163183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.163132    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:53.163278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.163206    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:53.163278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.163249    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:53.164959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.164923    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:53.165053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.165007    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:53.165096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.165052    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:24:53.165721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.165690    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" event={"ID":"28a281b3-e4e7-47ff-83b7-c34d7866ac19","Type":"ContainerStarted","Data":"885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06"}
Oct 30 14:24:53.244476 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.244450    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:24:53.245283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.245259    4530 patch_prober.go:29] interesting pod/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 14:24:53.245395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.245304    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 14:24:53.749405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:53.749374    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:24:54.176355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:54.176322    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" event={"ID":"3310a9c6-08cc-46ef-8a4a-27e960c28216","Type":"ContainerStarted","Data":"4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d"}
Oct 30 14:24:54.178445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:54.178415    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" event={"ID":"89261fc0-4ce9-4d2a-b886-f730e9424523","Type":"ContainerStarted","Data":"0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548"}
Oct 30 14:24:54.244564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:54.244512    4530 patch_prober.go:29] interesting pod/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 14:24:54.244987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:54.244599    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 14:24:54.463851 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:54.463757    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:24:54.465176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:54.465151    4530 patch_prober.go:29] interesting pod/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 14:24:54.465238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:54.465204    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 14:24:54.643151 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:54.643119    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:24:54.647739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:54.647706    4530 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 14:24:54.648010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:54.647978    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 14:24:54.765049 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:54.764967    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:24:55.278247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:55.278214    4530 patch_prober.go:29] interesting pod/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 14:24:55.279160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:55.279121    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 14:24:55.468562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:55.468492    4530 patch_prober.go:29] interesting pod/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 14:24:55.469106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:55.468842    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 14:24:55.647567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:55.646092    4530 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 14:24:55.647567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:55.646152    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 14:24:56.250995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:56.250957    4530 patch_prober.go:29] interesting pod/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 14:24:56.251332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:56.251280    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 14:24:56.464449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:56.464407    4530 patch_prober.go:29] interesting pod/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 14:24:56.464985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:56.464460    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 14:24:56.642018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:56.641976    4530 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 14:24:56.642345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:56.642313    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 14:24:57.032739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:57.032651    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:24:57.254569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:57.251947    4530 patch_prober.go:29] interesting pod/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 14:24:57.254569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:57.252015    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 14:24:57.464625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:57.464021    4530 patch_prober.go:29] interesting pod/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 14:24:57.464625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:57.464071    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 14:24:57.640566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:57.640043    4530 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 14:24:57.640566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:57.640105    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 14:24:58.254566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:58.251223    4530 patch_prober.go:29] interesting pod/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 14:24:58.254566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:58.251283    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 14:24:58.464193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:58.464155    4530 patch_prober.go:29] interesting pod/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 14:24:58.464368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:58.464207    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 14:24:58.637966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:58.637924    4530 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 14:24:58.638613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:58.638578    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 14:24:59.252987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:59.252949    4530 patch_prober.go:29] interesting pod/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 14:24:59.253316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:59.253286    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 14:24:59.464675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:59.464636    4530 patch_prober.go:29] interesting pod/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 14:24:59.464850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:59.464705    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 14:24:59.637124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:59.637034    4530 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 14:24:59.637124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:24:59.637101    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 14:25:00.250860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:00.250822    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:25:00.251756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:00.251738    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:25:00.463812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:00.463767    4530 patch_prober.go:29] interesting pod/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 14:25:00.464032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:00.463837    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 14:25:00.644382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:00.644348    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:25:00.644873 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:00.644833    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:25:00.645521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:00.645497    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:25:01.469134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:01.469103    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:25:01.470258 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:01.470239    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:25:03.569665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:03.569636    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:25:05.524040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.523998    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"]
Oct 30 14:25:05.533335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.533303    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.549532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.545802    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"]
Oct 30 14:25:05.562770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.562744    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-648d5f\"/\"openshift-service-ca.crt\""
Oct 30 14:25:05.575641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.575597    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-648d5f\"/\"kube-root-ca.crt\""
Oct 30 14:25:05.575641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.575625    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-648d5f\"/\"test-model-648d5f-kserve-self-signed-certs\""
Oct 30 14:25:05.576467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.576449    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-648d5f\"/\"default-dockercfg-knfm6\""
Oct 30 14:25:05.650786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.650748    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-dshm\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.651127 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.651089    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31152392-0c90-49cc-8365-14ffd1037fb7-tls-certs\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.651379 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.651357    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-model-cache\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.651725 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.651705    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-home\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.651944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.651925    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-tmp-dir\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.652102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.652084    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5twtd\" (UniqueName: \"kubernetes.io/projected/31152392-0c90-49cc-8365-14ffd1037fb7-kube-api-access-5twtd\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.753295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.753258    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-tmp-dir\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.753494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.753303    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5twtd\" (UniqueName: \"kubernetes.io/projected/31152392-0c90-49cc-8365-14ffd1037fb7-kube-api-access-5twtd\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.753494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.753358    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-dshm\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.753494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.753389    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31152392-0c90-49cc-8365-14ffd1037fb7-tls-certs\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.753494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.753448    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-model-cache\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.753810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.753662    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-home\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.753866 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.753802    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-tmp-dir\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.753925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.753901    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-model-cache\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.753989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.753968    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-home\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.755696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.755674    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-dshm\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.756055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.756037    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31152392-0c90-49cc-8365-14ffd1037fb7-tls-certs\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.804917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.803751    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5twtd\" (UniqueName: \"kubernetes.io/projected/31152392-0c90-49cc-8365-14ffd1037fb7-kube-api-access-5twtd\") pod \"test-model-648d5f-kserve-646847897c-zfbsj\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") " pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:05.858564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:05.858334    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:06.433719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:06.433623    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"]
Oct 30 14:25:07.457624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:07.457573    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj" event={"ID":"31152392-0c90-49cc-8365-14ffd1037fb7","Type":"ContainerStarted","Data":"305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a"}
Oct 30 14:25:07.457624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:07.457636    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj" event={"ID":"31152392-0c90-49cc-8365-14ffd1037fb7","Type":"ContainerStarted","Data":"85c5a2a46ab14977d58b25fda91a88daed69bdb6592f86ec833902ac463ca84c"}
Oct 30 14:25:07.534570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:07.529362    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj" podStartSLOduration=2.5293395800000003 podStartE2EDuration="2.52933958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:07.525914498 +0000 UTC m=+2172.557813439" watchObservedRunningTime="2026-10-30 14:25:07.52933958 +0000 UTC m=+2172.561238523"
Oct 30 14:25:07.875469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:07.875435    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:11.370637 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:25:11.370597    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:25:15.552955 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:15.552921    4530 generic.go:356] "Generic (PLEG): container finished" podID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerID="14b2a068f659197422cca734f2d5a9cf8ae07da516bad24d189aabe2f4be8445" exitCode=0
Oct 30 14:25:15.554270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:15.552988    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" event={"ID":"a83aae0b-4976-4a91-937c-c1e3dda7f151","Type":"ContainerDied","Data":"14b2a068f659197422cca734f2d5a9cf8ae07da516bad24d189aabe2f4be8445"}
Oct 30 14:25:15.554270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:15.553452    4530 scope.go:122] "RemoveContainer" containerID="14b2a068f659197422cca734f2d5a9cf8ae07da516bad24d189aabe2f4be8445"
Oct 30 14:25:15.555334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:15.555302    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:25:15.555410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:15.555372    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:25:15.555465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:15.555433    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:25:15.875169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:15.875084    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:15.887559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:15.887518    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:16.572565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:16.569038    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" event={"ID":"a83aae0b-4976-4a91-937c-c1e3dda7f151","Type":"ContainerStarted","Data":"ee7fc65c1cc0cede66d222bcd7d1f5418082bd89ad06834169913ce4e14b0c35"}
Oct 30 14:25:16.578821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:16.574139    4530 generic.go:356] "Generic (PLEG): container finished" podID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerID="487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312" exitCode=0
Oct 30 14:25:16.578821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:16.575533    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" event={"ID":"467f07a9-7aa2-4f27-9d2e-11106f3deb44","Type":"ContainerDied","Data":"487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312"}
Oct 30 14:25:16.578821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:16.576036    4530 scope.go:122] "RemoveContainer" containerID="487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312"
Oct 30 14:25:16.578821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:16.578318    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:25:16.578821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:16.578419    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:25:16.578821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:16.578466    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:25:16.612563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:16.604881    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:25:17.176589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:17.176557    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:25:17.178156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:17.178128    4530 patch_prober.go:29] interesting pod/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 14:25:17.178286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:17.178191    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 14:25:17.503623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:17.502642    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:25:17.637100 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:17.637066    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" event={"ID":"467f07a9-7aa2-4f27-9d2e-11106f3deb44","Type":"ContainerStarted","Data":"7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8"}
Oct 30 14:25:18.176350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:18.176305    4530 patch_prober.go:29] interesting pod/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 14:25:18.176532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:18.176380    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 14:25:18.515626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:18.514613    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:25:18.572865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:18.572810    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:25:18.573805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:18.573766    4530 patch_prober.go:29] interesting pod/e2e-inf-a-409364-openshift-default-868c776847-ntjrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 14:25:18.573944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:18.573827    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 14:25:19.176240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:19.176202    4530 patch_prober.go:29] interesting pod/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 14:25:19.176664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:19.176259    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 14:25:19.584565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:19.581900    4530 patch_prober.go:29] interesting pod/e2e-inf-a-409364-openshift-default-868c776847-ntjrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 14:25:19.584565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:19.581966    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 14:25:20.176867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:20.176826    4530 patch_prober.go:29] interesting pod/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 14:25:20.177280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:20.176912    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 14:25:20.579585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:20.575963    4530 patch_prober.go:29] interesting pod/e2e-inf-a-409364-openshift-default-868c776847-ntjrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 14:25:20.579585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:20.576049    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 14:25:21.176941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:21.176909    4530 patch_prober.go:29] interesting pod/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 14:25:21.177331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:21.176971    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 14:25:21.724165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:21.647666    4530 patch_prober.go:29] interesting pod/e2e-inf-a-409364-openshift-default-868c776847-ntjrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 14:25:21.724165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:21.647745    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 14:25:22.223085 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.157604    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"]
Oct 30 14:25:22.223085 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.178159    4530 patch_prober.go:29] interesting pod/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 14:25:22.223085 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.178213    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 14:25:22.255593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.226371    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.286711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.284407    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"]
Oct 30 14:25:22.306010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.305962    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fde75d-openshift-default-dockercfg-dp8mb\""
Oct 30 14:25:22.483510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.479820    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-envoy\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.509567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.483851    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-certs\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.511771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.511735    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-podinfo\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.512201 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.512173    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-credential-socket\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.512393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.512372    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e271438d-7e48-47fe-887b-d3ba07d2a875-istiod-ca-cert\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.512719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.512685    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-token\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.512965 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.512947    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-data\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.520467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.520439    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-socket\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.520680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.520659    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pkhm\" (UniqueName: \"kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-kube-api-access-9pkhm\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.608567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.573233    4530 patch_prober.go:29] interesting pod/e2e-inf-a-409364-openshift-default-868c776847-ntjrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 14:25:22.608567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.573301    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.629217    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-token\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.629293    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-data\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.629332    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-socket\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.629362    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pkhm\" (UniqueName: \"kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-kube-api-access-9pkhm\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.629451    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-envoy\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.629490    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-certs\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.629562    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-podinfo\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.629748    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-credential-socket\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.629785    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e271438d-7e48-47fe-887b-d3ba07d2a875-istiod-ca-cert\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.630869    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e271438d-7e48-47fe-887b-d3ba07d2a875-istiod-ca-cert\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.631502    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-data\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.631771    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-socket\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.646569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.646435    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-credential-socket\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.647410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.646653    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-certs\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.649900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.649872    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-podinfo\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.650093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.649882    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-envoy\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.658047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.658016    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-token\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.658414 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.658393    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pkhm\" (UniqueName: \"kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-kube-api-access-9pkhm\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:22.817010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:22.816976    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:23.216949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:23.216919    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"]
Oct 30 14:25:23.218797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:23.218747    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:25:23.218931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:23.218841    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:25:23.218931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:23.218884    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:25:23.380445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:23.379411    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:25:23.382414 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:23.382387    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:25:23.729175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:23.729145    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:25:23.730644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:23.730621    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:25:23.871593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:23.870112    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" event={"ID":"e271438d-7e48-47fe-887b-d3ba07d2a875","Type":"ContainerStarted","Data":"5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a"}
Oct 30 14:25:23.871593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:23.870176    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" event={"ID":"e271438d-7e48-47fe-887b-d3ba07d2a875","Type":"ContainerStarted","Data":"c2fcb01f4da372d3fcff83773a552b24f87df2949426ca14615794d7973263d8"}
Oct 30 14:25:23.923519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:23.923469    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" podStartSLOduration=1.9234546890000002 podStartE2EDuration="1.923454689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:23.922775334 +0000 UTC m=+2188.954674273" watchObservedRunningTime="2026-10-30 14:25:23.923454689 +0000 UTC m=+2188.955353628"
Oct 30 14:25:24.818087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:24.818049    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:24.824514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:24.824485    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:24.877114 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:24.877083    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:24.877988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:24.877959    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:25:28.478896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.478858    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"]
Oct 30 14:25:28.483207 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.483184    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.486292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.486254    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-certs\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.486426 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.486308    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt755\" (UniqueName: \"kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-kube-api-access-wt755\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.486502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.486433    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-data\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.486502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.486475    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-socket\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.486681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.486503    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istiod-ca-cert\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.486681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.486571    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-podinfo\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.486852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.486698    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-token\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.486852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.486756    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-credential-socket\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.486852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.486828    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-envoy\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.513349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.513320    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"]
Oct 30 14:25:28.552146 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.552118    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b9c4b4-openshift-default-dockercfg-c9bqk\""
Oct 30 14:25:28.587840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.587813    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt755\" (UniqueName: \"kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-kube-api-access-wt755\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.587974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.587913    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-data\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.587974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.587940    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-socket\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.587974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.587959    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istiod-ca-cert\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.588148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.587991    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-podinfo\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.588148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.588098    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-token\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.588266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.588155    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-credential-socket\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.588330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.588300    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-data\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.588395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.588323    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-envoy\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.588395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.588371    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-certs\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.588500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.588401    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-socket\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.588500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.588475    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-credential-socket\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.588716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.588697    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-certs\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.588786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.588729    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istiod-ca-cert\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.590484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.590463    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-envoy\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.590778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.590757    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-podinfo\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.617813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.617788    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-token\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.617982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.617965    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt755\" (UniqueName: \"kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-kube-api-access-wt755\") pod \"e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") " pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:28.805824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:28.805797    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:29.129650 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:25:29.129622    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06147d62_09b3_4e0e_bb3a_a8fad254a8dd.slice/crio-3a64bcb679814b74569a24414146882102cad22608cc9bf29e29a6c2b09b6517 WatchSource:0}: Error finding container 3a64bcb679814b74569a24414146882102cad22608cc9bf29e29a6c2b09b6517: Status 404 returned error can't find the container with id 3a64bcb679814b74569a24414146882102cad22608cc9bf29e29a6c2b09b6517
Oct 30 14:25:29.131675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:29.131612    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"]
Oct 30 14:25:29.132488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:29.132457    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:25:29.132578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:29.132522    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:25:29.132644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:29.132576    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:25:30.080910 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:30.080468    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" event={"ID":"06147d62-09b3-4e0e-bb3a-a8fad254a8dd","Type":"ContainerStarted","Data":"e7460e84dd32a353e5f78cbaa9bb41be362e6514c17d918359dea4465abc2dfc"}
Oct 30 14:25:30.081384 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:30.081036    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" event={"ID":"06147d62-09b3-4e0e-bb3a-a8fad254a8dd","Type":"ContainerStarted","Data":"3a64bcb679814b74569a24414146882102cad22608cc9bf29e29a6c2b09b6517"}
Oct 30 14:25:30.150738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:30.150680    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" podStartSLOduration=2.150651884 podStartE2EDuration="2.150651884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:30.14593964 +0000 UTC m=+2195.177838601" watchObservedRunningTime="2026-10-30 14:25:30.150651884 +0000 UTC m=+2195.182550824"
Oct 30 14:25:30.806280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:30.806248    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:30.812294 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:30.812269    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:31.088346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:31.088265    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:31.089327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:31.089301    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:25:35.743864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.743828    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"]
Oct 30 14:25:35.748256 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.748226    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:35.769378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.769356    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a9542c\"/\"kube-root-ca.crt\""
Oct 30 14:25:35.770604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.770588    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a9542c\"/\"openshift-service-ca.crt\""
Oct 30 14:25:35.792802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.791727    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a9542c\"/\"test-model-a9542c-kserve-self-signed-certs\""
Oct 30 14:25:35.792802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.791786    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a9542c\"/\"default-dockercfg-9d6xx\""
Oct 30 14:25:35.792802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.792005    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"]
Oct 30 14:25:35.883839 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.883798    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92c9a57-aa24-48a9-8278-cca33d6ad220-tls-certs\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:35.884004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.883898    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-home\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:35.884004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.883977    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-model-cache\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:35.884091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.884012    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bgz2\" (UniqueName: \"kubernetes.io/projected/b92c9a57-aa24-48a9-8278-cca33d6ad220-kube-api-access-5bgz2\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:35.884091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.884076    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-dshm\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:35.884292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.884275    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-tmp-dir\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.000611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.993823    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92c9a57-aa24-48a9-8278-cca33d6ad220-tls-certs\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.000611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.993901    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-home\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.000611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.993947    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-model-cache\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.000611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.993977    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bgz2\" (UniqueName: \"kubernetes.io/projected/b92c9a57-aa24-48a9-8278-cca33d6ad220-kube-api-access-5bgz2\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.000611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.994057    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-dshm\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.000611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.994269    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-tmp-dir\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.000611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:35.995051    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-tmp-dir\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.008571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:36.006012    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-model-cache\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.008571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:36.006434    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-home\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.031577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:36.024225    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-dshm\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.041561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:36.040363    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92c9a57-aa24-48a9-8278-cca33d6ad220-tls-certs\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.077572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:36.077464    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bgz2\" (UniqueName: \"kubernetes.io/projected/b92c9a57-aa24-48a9-8278-cca33d6ad220-kube-api-access-5bgz2\") pod \"test-model-a9542c-kserve-6595554bd7-bbvt5\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") " pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.362247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:36.362217    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:36.675474 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:25:36.675443    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb92c9a57_aa24_48a9_8278_cca33d6ad220.slice/crio-e2b41b2752b93f038547352efb0f8112d8fcbc6b93280a9d8500f828e11a4be8 WatchSource:0}: Error finding container e2b41b2752b93f038547352efb0f8112d8fcbc6b93280a9d8500f828e11a4be8: Status 404 returned error can't find the container with id e2b41b2752b93f038547352efb0f8112d8fcbc6b93280a9d8500f828e11a4be8
Oct 30 14:25:36.676629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:36.676605    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"]
Oct 30 14:25:37.170571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:37.170500    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5" event={"ID":"b92c9a57-aa24-48a9-8278-cca33d6ad220","Type":"ContainerStarted","Data":"793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e"}
Oct 30 14:25:37.170571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:37.170564    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5" event={"ID":"b92c9a57-aa24-48a9-8278-cca33d6ad220","Type":"ContainerStarted","Data":"e2b41b2752b93f038547352efb0f8112d8fcbc6b93280a9d8500f828e11a4be8"}
Oct 30 14:25:37.253147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:37.253080    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5" podStartSLOduration=2.253057597 podStartE2EDuration="2.253057597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:37.239397374 +0000 UTC m=+2202.271296346" watchObservedRunningTime="2026-10-30 14:25:37.253057597 +0000 UTC m=+2202.284956537"
Oct 30 14:25:37.370757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:37.370714    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:41.386130 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:25:41.386016    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:25:42.278245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:42.278192    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"]
Oct 30 14:25:46.362551 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:46.362514    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:46.375764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:46.375732    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:47.294874 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:47.294843    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:25:57.311614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.311560    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk" podUID="4a29c13d-89ea-45f2-b3fe-137f9db7ecd8" containerName="main" containerID="cri-o://7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2" gracePeriod=45
Oct 30 14:25:57.587996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.587972    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:25:57.689415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.689381    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-model-cache\") pod \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") "
Oct 30 14:25:57.689643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.689435    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-dshm\") pod \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") "
Oct 30 14:25:57.689643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.689530    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-home\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-home\") pod \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") "
Oct 30 14:25:57.689790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.689665    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tmp-dir\") pod \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") "
Oct 30 14:25:57.689790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.689696    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-kube-api-access-59s5l\" (UniqueName: \"kubernetes.io/projected/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-kube-api-access-59s5l\") pod \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") "
Oct 30 14:25:57.689790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.689692    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-model-cache" pod "4a29c13d-89ea-45f2-b3fe-137f9db7ecd8" (UID: "4a29c13d-89ea-45f2-b3fe-137f9db7ecd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:25:57.689790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.689756    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tls-certs\") pod \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\" (UID: \"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8\") "
Oct 30 14:25:57.689790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.689763    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-home" pod "4a29c13d-89ea-45f2-b3fe-137f9db7ecd8" (UID: "4a29c13d-89ea-45f2-b3fe-137f9db7ecd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:25:57.690061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.689921    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tmp-dir" pod "4a29c13d-89ea-45f2-b3fe-137f9db7ecd8" (UID: "4a29c13d-89ea-45f2-b3fe-137f9db7ecd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:25:57.690414 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.690391    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:25:57.690531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.690415    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:25:57.690531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.690430    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:25:57.691834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.691813    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-dshm" pod "4a29c13d-89ea-45f2-b3fe-137f9db7ecd8" (UID: "4a29c13d-89ea-45f2-b3fe-137f9db7ecd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:25:57.691948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.691836    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tls-certs" pod "4a29c13d-89ea-45f2-b3fe-137f9db7ecd8" (UID: "4a29c13d-89ea-45f2-b3fe-137f9db7ecd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:25:57.691948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.691870    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-kube-api-access-59s5l" pod "4a29c13d-89ea-45f2-b3fe-137f9db7ecd8" (UID: "4a29c13d-89ea-45f2-b3fe-137f9db7ecd8"). InnerVolumeSpecName "kube-api-access-59s5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:25:57.791902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.791877    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59s5l\" (UniqueName: \"kubernetes.io/projected/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-kube-api-access-59s5l\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:25:57.791902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.791903    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:25:57.792069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:57.791917    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:25:58.372630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:58.372588    4530 generic.go:356] "Generic (PLEG): container finished" podID="4a29c13d-89ea-45f2-b3fe-137f9db7ecd8" containerID="7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2" exitCode=0
Oct 30 14:25:58.373118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:58.372648    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"
Oct 30 14:25:58.373118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:58.372676    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk" event={"ID":"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8","Type":"ContainerDied","Data":"7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2"}
Oct 30 14:25:58.373118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:58.372728    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk" event={"ID":"4a29c13d-89ea-45f2-b3fe-137f9db7ecd8","Type":"ContainerDied","Data":"0fbda24ec248f049b3e24f320775a64cf9393c22615ef83466cd8faff5a4f7e5"}
Oct 30 14:25:58.373118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:58.372751    4530 scope.go:122] "RemoveContainer" containerID="7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2"
Oct 30 14:25:58.387702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:58.387347    4530 scope.go:122] "RemoveContainer" containerID="7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2"
Oct 30 14:25:58.387817 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:25:58.387787    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2\": container with ID starting with 7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2 not found: ID does not exist" containerID="7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2"
Oct 30 14:25:58.387926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:58.387827    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2"} err="failed to get container status \"7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2\": rpc error: code = NotFound desc = could not find container \"7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2\": container with ID starting with 7a5d206cc01248d587cc583561758b0ca0d386bca4f4271ea2c008d1af190ad2 not found: ID does not exist"
Oct 30 14:25:58.436251 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:58.436226    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"]
Oct 30 14:25:58.449831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:58.449811    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5a9d7b/test-model-5a9d7b-kserve-65c86c6dd7-x8nwk"]
Oct 30 14:25:59.116765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:25:59.116732    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a29c13d-89ea-45f2-b3fe-137f9db7ecd8" path="/var/lib/kubelet/pods/4a29c13d-89ea-45f2-b3fe-137f9db7ecd8/volumes"
Oct 30 14:26:04.424564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:04.421120    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:26:05.435714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:05.435670    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"]
Oct 30 14:26:05.436215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:05.435916    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd" podUID="ae78c3f3-873c-480c-9624-a27660a9f541" containerName="maas-api" containerID="cri-o://cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b" gracePeriod=30
Oct 30 14:26:06.095563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.091761    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:26:06.301781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.300293    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae78c3f3-873c-480c-9624-a27660a9f541-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae78c3f3-873c-480c-9624-a27660a9f541-openshift-service-ca\") pod \"ae78c3f3-873c-480c-9624-a27660a9f541\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") "
Oct 30 14:26:06.301781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.300362    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-metrics-certs\") pod \"ae78c3f3-873c-480c-9624-a27660a9f541\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") "
Oct 30 14:26:06.301781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.300606    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-maas-api-tls\") pod \"ae78c3f3-873c-480c-9624-a27660a9f541\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") "
Oct 30 14:26:06.301781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.300690    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae78c3f3-873c-480c-9624-a27660a9f541-kube-api-access-6qx6s\" (UniqueName: \"kubernetes.io/projected/ae78c3f3-873c-480c-9624-a27660a9f541-kube-api-access-6qx6s\") pod \"ae78c3f3-873c-480c-9624-a27660a9f541\" (UID: \"ae78c3f3-873c-480c-9624-a27660a9f541\") "
Oct 30 14:26:06.304090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.303240    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae78c3f3-873c-480c-9624-a27660a9f541-openshift-service-ca" pod "ae78c3f3-873c-480c-9624-a27660a9f541" (UID: "ae78c3f3-873c-480c-9624-a27660a9f541"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:26:06.307403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.307377    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae78c3f3-873c-480c-9624-a27660a9f541-kube-api-access-6qx6s" pod "ae78c3f3-873c-480c-9624-a27660a9f541" (UID: "ae78c3f3-873c-480c-9624-a27660a9f541"). InnerVolumeSpecName "kube-api-access-6qx6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:06.324561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.320450    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-maas-api-tls" pod "ae78c3f3-873c-480c-9624-a27660a9f541" (UID: "ae78c3f3-873c-480c-9624-a27660a9f541"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:26:06.324561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.320553    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-metrics-certs" pod "ae78c3f3-873c-480c-9624-a27660a9f541" (UID: "ae78c3f3-873c-480c-9624-a27660a9f541"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:26:06.495811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.440206    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae78c3f3-873c-480c-9624-a27660a9f541-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:06.495811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.480650    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:06.495811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.480671    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae78c3f3-873c-480c-9624-a27660a9f541-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:06.495811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.480685    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qx6s\" (UniqueName: \"kubernetes.io/projected/ae78c3f3-873c-480c-9624-a27660a9f541-kube-api-access-6qx6s\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:06.588621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.573237    4530 generic.go:356] "Generic (PLEG): container finished" podID="ae78c3f3-873c-480c-9624-a27660a9f541" containerID="cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b" exitCode=0
Oct 30 14:26:06.588621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.573522    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd" event={"ID":"ae78c3f3-873c-480c-9624-a27660a9f541","Type":"ContainerDied","Data":"cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b"}
Oct 30 14:26:06.588621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.573599    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd" event={"ID":"ae78c3f3-873c-480c-9624-a27660a9f541","Type":"ContainerDied","Data":"10a66b7287e5df5293c96c79f2b8aa15c5df3ca2ba24596a0ca4ba15ae52c5a4"}
Oct 30 14:26:06.588621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.573642    4530 scope.go:122] "RemoveContainer" containerID="cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b"
Oct 30 14:26:06.588621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.573849    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"
Oct 30 14:26:06.705575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.702662    4530 scope.go:122] "RemoveContainer" containerID="cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b"
Oct 30 14:26:06.705575 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:26:06.703073    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b\": container with ID starting with cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b not found: ID does not exist" containerID="cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b"
Oct 30 14:26:06.705575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.703103    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b"} err="failed to get container status \"cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b\": rpc error: code = NotFound desc = could not find container \"cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b\": container with ID starting with cc8f18ec074cffc9fea8593880ebd86a607694a19b532a110e45dcfbda8a052b not found: ID does not exist"
Oct 30 14:26:06.779569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.779084    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"]
Oct 30 14:26:06.843619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:06.835993    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea9ed2-66b6bf66f-49hvd"]
Oct 30 14:26:07.133617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:07.133261    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae78c3f3-873c-480c-9624-a27660a9f541" path="/var/lib/kubelet/pods/ae78c3f3-873c-480c-9624-a27660a9f541/volumes"
Oct 30 14:26:08.153513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:08.153484    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"]
Oct 30 14:26:11.368654 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:26:11.368617    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:26:13.976085 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:13.976047    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"]
Oct 30 14:26:13.976991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:13.976970    4530 cpu_manager.go:414] "containerMap: removing container" podUID="4a29c13d-89ea-45f2-b3fe-137f9db7ecd8" containerName="main"
Oct 30 14:26:13.976991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:13.976991    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a29c13d-89ea-45f2-b3fe-137f9db7ecd8" containerName="main"
Oct 30 14:26:13.977124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:13.977047    4530 cpu_manager.go:414] "containerMap: removing container" podUID="ae78c3f3-873c-480c-9624-a27660a9f541" containerName="maas-api"
Oct 30 14:26:13.977124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:13.977053    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae78c3f3-873c-480c-9624-a27660a9f541" containerName="maas-api"
Oct 30 14:26:13.977205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:13.977172    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae78c3f3-873c-480c-9624-a27660a9f541" containerName="maas-api"
Oct 30 14:26:13.977205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:13.977184    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a29c13d-89ea-45f2-b3fe-137f9db7ecd8" containerName="main"
Oct 30 14:26:13.982494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:13.982473    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.008584 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.001815    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"]
Oct 30 14:26:14.012969 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.012943    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0b1c15-serving-cert\""
Oct 30 14:26:14.078255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.078215    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.078456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.078334    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.078456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.078410    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-metrics-certs\") pod \"maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.078576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.078461    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7jr\" (UniqueName: \"kubernetes.io/projected/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-kube-api-access-vb7jr\") pod \"maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.179887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.179843    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.180092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.180017    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-metrics-certs\") pod \"maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.180174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.180094    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7jr\" (UniqueName: \"kubernetes.io/projected/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-kube-api-access-vb7jr\") pod \"maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.181383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.180592    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.181932 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.181887    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.183412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.183391    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-metrics-certs\") pod \"maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.183624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.183602    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.246932 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.246774    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7jr\" (UniqueName: \"kubernetes.io/projected/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-kube-api-access-vb7jr\") pod \"maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.323194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.322903    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:14.598223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.598185    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"]
Oct 30 14:26:14.598448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.598421    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerName="istio-proxy" containerID="cri-o://eea5cfc436f8fc748ac09214438dbae2e738ce3aee9f5a08be6c35dca8226025" gracePeriod=30
Oct 30 14:26:14.619170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.619141    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"]
Oct 30 14:26:14.619410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.619357    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy" containerID="cri-o://4b46e3491636130b863d20b3cdd6fcd6351dd192595fb0cf57b66d3a659b63f1" gracePeriod=30
Oct 30 14:26:14.624913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.624888    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"]
Oct 30 14:26:14.625083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.625063    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" podUID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" containerName="istio-proxy" containerID="cri-o://10e6e3484a2613c41c0067ddf930c38793b8c945460f7421b21b79f3e3fffe97" gracePeriod=30
Oct 30 14:26:14.629039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.629016    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"]
Oct 30 14:26:14.629193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.629164    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" podUID="d61c8db3-2589-43d8-85a0-c49eccfe6023" containerName="istio-proxy" containerID="cri-o://7efc98a313c8ff23993b4af8dff369cb9f8fbfba491ba05cada609b2b6499f63" gracePeriod=30
Oct 30 14:26:14.674015 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:26:14.673985    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b35d7cc_e2d8_4595_9d9b_06b4be0165db.slice/crio-e083484b7783c0fbb7c35b552f92a235c94c374216e547269804cefe5762ee2e WatchSource:0}: Error finding container e083484b7783c0fbb7c35b552f92a235c94c374216e547269804cefe5762ee2e: Status 404 returned error can't find the container with id e083484b7783c0fbb7c35b552f92a235c94c374216e547269804cefe5762ee2e
Oct 30 14:26:14.678532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.677054    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"]
Oct 30 14:26:14.827562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.821938    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:26:14.827562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.822028    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" podUID="d61c8db3-2589-43d8-85a0-c49eccfe6023" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:26:14.852455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.852361    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657" event={"ID":"9b35d7cc-e2d8-4595-9d9b-06b4be0165db","Type":"ContainerStarted","Data":"e083484b7783c0fbb7c35b552f92a235c94c374216e547269804cefe5762ee2e"}
Oct 30 14:26:14.933400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.933360    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:26:14.933574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:14.933430    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" podUID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:26:15.150921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:15.150895    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:26:15.151288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:15.150940    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:26:15.863615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:15.863527    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657" event={"ID":"9b35d7cc-e2d8-4595-9d9b-06b4be0165db","Type":"ContainerStarted","Data":"0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3"}
Oct 30 14:26:15.863795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:15.863628    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:15.931617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:15.931567    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657" podStartSLOduration=2.503845259 podStartE2EDuration="2.93155216s" totalImagesPullingTime="427.706901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:15.927927686 +0000 UTC m=+2240.959826622" watchObservedRunningTime="2026-10-30 14:26:15.93155216 +0000 UTC m=+2240.963451089"
Oct 30 14:26:16.036329 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.036294    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"]
Oct 30 14:26:16.056707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.056557    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.079922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.075503    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fde75d-serving-cert\""
Oct 30 14:26:16.091864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.091836    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"]
Oct 30 14:26:16.226064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.225974    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-metrics-certs\") pod \"maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.226437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.226102    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c913bf32-ca38-48b7-98ec-b4ed589ab093-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.226437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.226183    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-maas-api-tls\") pod \"maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.226437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.226320    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq8fc\" (UniqueName: \"kubernetes.io/projected/c913bf32-ca38-48b7-98ec-b4ed589ab093-kube-api-access-kq8fc\") pod \"maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.332562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.331689    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-metrics-certs\") pod \"maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.358563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.357635    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c913bf32-ca38-48b7-98ec-b4ed589ab093-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.358563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.357742    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-maas-api-tls\") pod \"maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.358563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.357856    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq8fc\" (UniqueName: \"kubernetes.io/projected/c913bf32-ca38-48b7-98ec-b4ed589ab093-kube-api-access-kq8fc\") pod \"maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.358563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.358491    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c913bf32-ca38-48b7-98ec-b4ed589ab093-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.362563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.361700    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-maas-api-tls\") pod \"maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.382171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.369183    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-metrics-certs\") pod \"maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.401975 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.398302    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq8fc\" (UniqueName: \"kubernetes.io/projected/c913bf32-ca38-48b7-98ec-b4ed589ab093-kube-api-access-kq8fc\") pod \"maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.425564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.424424    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:16.867037 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:26:16.866998    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc913bf32_ca38_48b7_98ec_b4ed589ab093.slice/crio-79374ffada244f9ecc20b964c850962ebf781f65c693f9508031c27de3069c51 WatchSource:0}: Error finding container 79374ffada244f9ecc20b964c850962ebf781f65c693f9508031c27de3069c51: Status 404 returned error can't find the container with id 79374ffada244f9ecc20b964c850962ebf781f65c693f9508031c27de3069c51
Oct 30 14:26:16.869045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:16.869019    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"]
Oct 30 14:26:17.593481 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:17.590650    4530 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:26:17.593481 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:17.590734    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:26:17.888319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:17.888223    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw" event={"ID":"c913bf32-ca38-48b7-98ec-b4ed589ab093","Type":"ContainerStarted","Data":"e65516694a12afda3270e99d269680117f1285ee46035fab3a98df7c7404b7a5"}
Oct 30 14:26:17.888319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:17.888282    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw" event={"ID":"c913bf32-ca38-48b7-98ec-b4ed589ab093","Type":"ContainerStarted","Data":"79374ffada244f9ecc20b964c850962ebf781f65c693f9508031c27de3069c51"}
Oct 30 14:26:17.888319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:17.888313    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:17.889992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:17.889965    4530 generic.go:356] "Generic (PLEG): container finished" podID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerID="2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f" exitCode=0
Oct 30 14:26:17.890110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:17.890035    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" event={"ID":"acdd5958-7f02-448b-8bcc-654433ec1cd2","Type":"ContainerDied","Data":"2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f"}
Oct 30 14:26:17.890452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:17.890437    4530 scope.go:122] "RemoveContainer" containerID="2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f"
Oct 30 14:26:17.892186 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:17.892157    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:26:17.892240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:17.892228    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:26:17.892276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:17.892260    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:26:17.937755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:17.937715    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw" podStartSLOduration=2.593275286 podStartE2EDuration="2.937702515s" totalImagesPullingTime="344.427229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:17.936450363 +0000 UTC m=+2242.968349301" watchObservedRunningTime="2026-10-30 14:26:17.937702515 +0000 UTC m=+2242.969601453"
Oct 30 14:26:18.924984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:18.924944    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" event={"ID":"acdd5958-7f02-448b-8bcc-654433ec1cd2","Type":"ContainerStarted","Data":"4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d"}
Oct 30 14:26:19.334610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:19.334574    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:26:19.335794 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:19.335770    4530 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 14:26:19.335896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:19.335819    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 14:26:19.940489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:19.940462    4530 generic.go:356] "Generic (PLEG): container finished" podID="d61c8db3-2589-43d8-85a0-c49eccfe6023" containerID="7efc98a313c8ff23993b4af8dff369cb9f8fbfba491ba05cada609b2b6499f63" exitCode=0
Oct 30 14:26:19.940919 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:19.940571    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" event={"ID":"d61c8db3-2589-43d8-85a0-c49eccfe6023","Type":"ContainerDied","Data":"7efc98a313c8ff23993b4af8dff369cb9f8fbfba491ba05cada609b2b6499f63"}
Oct 30 14:26:19.942632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:19.942610    4530 generic.go:356] "Generic (PLEG): container finished" podID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" containerID="10e6e3484a2613c41c0067ddf930c38793b8c945460f7421b21b79f3e3fffe97" exitCode=0
Oct 30 14:26:19.942754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:19.942684    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" event={"ID":"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0","Type":"ContainerDied","Data":"10e6e3484a2613c41c0067ddf930c38793b8c945460f7421b21b79f3e3fffe97"}
Oct 30 14:26:19.945190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:19.945171    4530 generic.go:356] "Generic (PLEG): container finished" podID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerID="4b46e3491636130b863d20b3cdd6fcd6351dd192595fb0cf57b66d3a659b63f1" exitCode=0
Oct 30 14:26:19.945724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:19.945354    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" event={"ID":"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8","Type":"ContainerDied","Data":"4b46e3491636130b863d20b3cdd6fcd6351dd192595fb0cf57b66d3a659b63f1"}
Oct 30 14:26:19.945724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:19.945407    4530 scope.go:122] "RemoveContainer" containerID="fe41117d5c88dfbd418f289642ac23ef1824e9ec78a41596d91400ff4a4d5181"
Oct 30 14:26:19.947501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:19.947478    4530 generic.go:356] "Generic (PLEG): container finished" podID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerID="eea5cfc436f8fc748ac09214438dbae2e738ce3aee9f5a08be6c35dca8226025" exitCode=0
Oct 30 14:26:19.947868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:19.947834    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" event={"ID":"d8605fd0-853e-4cb6-9906-ceb6a07f8430","Type":"ContainerDied","Data":"eea5cfc436f8fc748ac09214438dbae2e738ce3aee9f5a08be6c35dca8226025"}
Oct 30 14:26:20.022707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.021337    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:26:20.135084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.135061    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:26:20.140674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.140232    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-envoy\") pod \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") "
Oct 30 14:26:20.140674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.140287    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-data\") pod \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") "
Oct 30 14:26:20.140674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.140338    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-socket\") pod \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") "
Oct 30 14:26:20.140674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.140408    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-credential-socket\") pod \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") "
Oct 30 14:26:20.140674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.140494    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istiod-ca-cert\") pod \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") "
Oct 30 14:26:20.141023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.140924    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-data" pod "d8605fd0-853e-4cb6-9906-ceb6a07f8430" (UID: "d8605fd0-853e-4cb6-9906-ceb6a07f8430"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.141087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.141036    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-socket" pod "d8605fd0-853e-4cb6-9906-ceb6a07f8430" (UID: "d8605fd0-853e-4cb6-9906-ceb6a07f8430"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.144735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.141280    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-credential-socket" pod "d8605fd0-853e-4cb6-9906-ceb6a07f8430" (UID: "d8605fd0-853e-4cb6-9906-ceb6a07f8430"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.144735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.141405    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-podinfo\") pod \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") "
Oct 30 14:26:20.144735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.141588    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-token\" (UniqueName: \"kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-token\") pod \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") "
Oct 30 14:26:20.144735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.141770    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-kube-api-access-pj6ph\" (UniqueName: \"kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-kube-api-access-pj6ph\") pod \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") "
Oct 30 14:26:20.144735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.141778    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istiod-ca-cert" pod "d8605fd0-853e-4cb6-9906-ceb6a07f8430" (UID: "d8605fd0-853e-4cb6-9906-ceb6a07f8430"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:26:20.144735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.141817    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-certs\") pod \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\" (UID: \"d8605fd0-853e-4cb6-9906-ceb6a07f8430\") "
Oct 30 14:26:20.146106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.145606    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.146106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.145646    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.146106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.145663    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.146106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.145677    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.146771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.146436    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-certs" pod "d8605fd0-853e-4cb6-9906-ceb6a07f8430" (UID: "d8605fd0-853e-4cb6-9906-ceb6a07f8430"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.149658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.149630    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-token" pod "d8605fd0-853e-4cb6-9906-ceb6a07f8430" (UID: "d8605fd0-853e-4cb6-9906-ceb6a07f8430"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:20.154217 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.154157    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-kube-api-access-pj6ph" pod "d8605fd0-853e-4cb6-9906-ceb6a07f8430" (UID: "d8605fd0-853e-4cb6-9906-ceb6a07f8430"). InnerVolumeSpecName "kube-api-access-pj6ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:20.189022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.188724    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:26:20.265648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.259769    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-envoy" pod "d8605fd0-853e-4cb6-9906-ceb6a07f8430" (UID: "d8605fd0-853e-4cb6-9906-ceb6a07f8430"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.267121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.267083    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.267502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.267132    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj6ph\" (UniqueName: \"kubernetes.io/projected/d8605fd0-853e-4cb6-9906-ceb6a07f8430-kube-api-access-pj6ph\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.267502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.267149    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.267502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.267164    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.277200 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.277166    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-podinfo" pod "d8605fd0-853e-4cb6-9906-ceb6a07f8430" (UID: "d8605fd0-853e-4cb6-9906-ceb6a07f8430"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:26:20.279370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.279237    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:26:20.316231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.316206    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:26:20.335164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.335137    4530 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 14:26:20.335267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.335192    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 14:26:20.368186 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368155    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-data\") pod \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") "
Oct 30 14:26:20.368377 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368362    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-envoy\") pod \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") "
Oct 30 14:26:20.368465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368398    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61c8db3-2589-43d8-85a0-c49eccfe6023-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61c8db3-2589-43d8-85a0-c49eccfe6023-istiod-ca-cert\") pod \"d61c8db3-2589-43d8-85a0-c49eccfe6023\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") "
Oct 30 14:26:20.368465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368430    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-envoy\") pod \"d61c8db3-2589-43d8-85a0-c49eccfe6023\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") "
Oct 30 14:26:20.368465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368440    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-data" pod "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" (UID: "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.368681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368475    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-kube-api-access-nsznk\" (UniqueName: \"kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-kube-api-access-nsznk\") pod \"d61c8db3-2589-43d8-85a0-c49eccfe6023\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") "
Oct 30 14:26:20.368681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368510    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-token\" (UniqueName: \"kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-token\") pod \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") "
Oct 30 14:26:20.368681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368554    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-kube-api-access-l942p\" (UniqueName: \"kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-kube-api-access-l942p\") pod \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") "
Oct 30 14:26:20.368681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368618    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-socket\") pod \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") "
Oct 30 14:26:20.368681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368645    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-podinfo\") pod \"d61c8db3-2589-43d8-85a0-c49eccfe6023\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") "
Oct 30 14:26:20.368966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368689    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-podinfo\") pod \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") "
Oct 30 14:26:20.368966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368737    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-credential-socket\") pod \"d61c8db3-2589-43d8-85a0-c49eccfe6023\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") "
Oct 30 14:26:20.368966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368774    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istiod-ca-cert\") pod \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") "
Oct 30 14:26:20.368966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368816    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istiod-ca-cert\") pod \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") "
Oct 30 14:26:20.368966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368845    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-credential-socket\") pod \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") "
Oct 30 14:26:20.368966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368900    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-token\" (UniqueName: \"kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-token\") pod \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") "
Oct 30 14:26:20.368966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368953    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-socket\") pod \"d61c8db3-2589-43d8-85a0-c49eccfe6023\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") "
Oct 30 14:26:20.369343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368963    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-socket" pod "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" (UID: "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.369343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.368984    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-certs\") pod \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\" (UID: \"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8\") "
Oct 30 14:26:20.369343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.369109    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-data\") pod \"d61c8db3-2589-43d8-85a0-c49eccfe6023\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") "
Oct 30 14:26:20.369343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.369158    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-envoy\") pod \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") "
Oct 30 14:26:20.369343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.369196    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-token\" (UniqueName: \"kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-token\") pod \"d61c8db3-2589-43d8-85a0-c49eccfe6023\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") "
Oct 30 14:26:20.369343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.369267    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-kube-api-access-w8bcc\" (UniqueName: \"kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-kube-api-access-w8bcc\") pod \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") "
Oct 30 14:26:20.369343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.369335    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-socket\") pod \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") "
Oct 30 14:26:20.369755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.369367    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-credential-socket\") pod \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") "
Oct 30 14:26:20.369755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.369398    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-podinfo\") pod \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") "
Oct 30 14:26:20.369755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.369444    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-certs\") pod \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") "
Oct 30 14:26:20.371413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.369987    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istiod-ca-cert" pod "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" (UID: "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:26:20.371413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.370210    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-certs" pod "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" (UID: "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.371413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.370409    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-certs" pod "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" (UID: "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.371413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.370508    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istiod-ca-cert" pod "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" (UID: "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:26:20.371413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.370768    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-credential-socket" pod "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" (UID: "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.371413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.370969    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-data" pod "d61c8db3-2589-43d8-85a0-c49eccfe6023" (UID: "d61c8db3-2589-43d8-85a0-c49eccfe6023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.371916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.371510    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61c8db3-2589-43d8-85a0-c49eccfe6023-istiod-ca-cert" pod "d61c8db3-2589-43d8-85a0-c49eccfe6023" (UID: "d61c8db3-2589-43d8-85a0-c49eccfe6023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:26:20.371916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.371773    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-socket" pod "d61c8db3-2589-43d8-85a0-c49eccfe6023" (UID: "d61c8db3-2589-43d8-85a0-c49eccfe6023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.371916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.371891    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-envoy" pod "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" (UID: "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.372430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372245    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-credential-socket" pod "d61c8db3-2589-43d8-85a0-c49eccfe6023" (UID: "d61c8db3-2589-43d8-85a0-c49eccfe6023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.372534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372467    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.372534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372490    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.372534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372504    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.372534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372517    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.372534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372529    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.372831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372567    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.372831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372584    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.372831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372597    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.372831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372609    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.372831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372622    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.372831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372635    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.372831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372648    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61c8db3-2589-43d8-85a0-c49eccfe6023-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.372831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.372661    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8605fd0-853e-4cb6-9906-ceb6a07f8430-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.373289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.373057    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-credential-socket" pod "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" (UID: "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.373289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.373106    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-socket" pod "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" (UID: "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.373289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.373199    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-kube-api-access-l942p" pod "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" (UID: "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8"). InnerVolumeSpecName "kube-api-access-l942p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:20.375767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.375741    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-podinfo" pod "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" (UID: "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:26:20.375878 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.375828    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-envoy" pod "d61c8db3-2589-43d8-85a0-c49eccfe6023" (UID: "d61c8db3-2589-43d8-85a0-c49eccfe6023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.375959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.375930    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-token" pod "d61c8db3-2589-43d8-85a0-c49eccfe6023" (UID: "d61c8db3-2589-43d8-85a0-c49eccfe6023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:20.376276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.376142    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-envoy" pod "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" (UID: "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.376276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.376192    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-token" pod "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" (UID: "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:20.376276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.376248    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-token" pod "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" (UID: "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:20.376276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.376274    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-kube-api-access-nsznk" pod "d61c8db3-2589-43d8-85a0-c49eccfe6023" (UID: "d61c8db3-2589-43d8-85a0-c49eccfe6023"). InnerVolumeSpecName "kube-api-access-nsznk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:20.376814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.376783    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-kube-api-access-w8bcc" pod "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" (UID: "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0"). InnerVolumeSpecName "kube-api-access-w8bcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:20.377706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.377679    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-podinfo" pod "d61c8db3-2589-43d8-85a0-c49eccfe6023" (UID: "d61c8db3-2589-43d8-85a0-c49eccfe6023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:26:20.377920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.377893    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-podinfo" pod "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" (UID: "f4db2ee9-dd55-4d4a-89cc-f92c77d26db8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:26:20.473742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.473653    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-data\") pod \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\" (UID: \"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0\") "
Oct 30 14:26:20.473742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.473702    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-certs\") pod \"d61c8db3-2589-43d8-85a0-c49eccfe6023\" (UID: \"d61c8db3-2589-43d8-85a0-c49eccfe6023\") "
Oct 30 14:26:20.473996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.473945    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-data" pod "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" (UID: "cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.473996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.473966    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-certs" pod "d61c8db3-2589-43d8-85a0-c49eccfe6023" (UID: "d61c8db3-2589-43d8-85a0-c49eccfe6023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:20.474332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474314    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474334    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474344    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474352    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474360    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474369    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474377    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsznk\" (UniqueName: \"kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-kube-api-access-nsznk\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474385    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474393    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l942p\" (UniqueName: \"kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-kube-api-access-l942p\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474402    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474410    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474419    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474427    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474435    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61c8db3-2589-43d8-85a0-c49eccfe6023-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.474823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.474443    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8bcc\" (UniqueName: \"kubernetes.io/projected/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0-kube-api-access-w8bcc\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:20.956823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.956795    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"
Oct 30 14:26:20.957344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.956827    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf" event={"ID":"cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0","Type":"ContainerDied","Data":"a3bfe72efcdd013046719ecef323d735a455788aba33f1dfeb9e0d8d61f580e6"}
Oct 30 14:26:20.957344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.956904    4530 scope.go:122] "RemoveContainer" containerID="10e6e3484a2613c41c0067ddf930c38793b8c945460f7421b21b79f3e3fffe97"
Oct 30 14:26:20.958743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.958722    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"
Oct 30 14:26:20.958882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.958797    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg" event={"ID":"f4db2ee9-dd55-4d4a-89cc-f92c77d26db8","Type":"ContainerDied","Data":"389fc07b153ef468e0eb12b565b74fde075040d110f70f56a128abbd0ada5fb4"}
Oct 30 14:26:20.960517 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.960495    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"
Oct 30 14:26:20.960628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.960509    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7" event={"ID":"d8605fd0-853e-4cb6-9906-ceb6a07f8430","Type":"ContainerDied","Data":"ee8e0a750b27be66bb643f04ad0581c8ea9703ea20d4ccf5bcdbd2dc7bf33281"}
Oct 30 14:26:20.966050 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.966020    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"
Oct 30 14:26:20.966164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.966052    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5" event={"ID":"d61c8db3-2589-43d8-85a0-c49eccfe6023","Type":"ContainerDied","Data":"8d98dd679971b00be4222e1ac8614ff27d5156e4cc5b21c05af4ddd4090752f6"}
Oct 30 14:26:20.971466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:20.971441    4530 scope.go:122] "RemoveContainer" containerID="4b46e3491636130b863d20b3cdd6fcd6351dd192595fb0cf57b66d3a659b63f1"
Oct 30 14:26:21.004031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.000516    4530 scope.go:122] "RemoveContainer" containerID="eea5cfc436f8fc748ac09214438dbae2e738ce3aee9f5a08be6c35dca8226025"
Oct 30 14:26:21.015076 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.015051    4530 scope.go:122] "RemoveContainer" containerID="7efc98a313c8ff23993b4af8dff369cb9f8fbfba491ba05cada609b2b6499f63"
Oct 30 14:26:21.024266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.024243    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"]
Oct 30 14:26:21.032570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.032535    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-wcsg7"]
Oct 30 14:26:21.073843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.073822    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"]
Oct 30 14:26:21.078053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.078032    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-x7wtf"]
Oct 30 14:26:21.121528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.121496    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" path="/var/lib/kubelet/pods/cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0/volumes"
Oct 30 14:26:21.122051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.122033    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" path="/var/lib/kubelet/pods/d8605fd0-853e-4cb6-9906-ceb6a07f8430/volumes"
Oct 30 14:26:21.130222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.130195    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"]
Oct 30 14:26:21.138415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.138373    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-zrtbg"]
Oct 30 14:26:21.210618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.205908    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"]
Oct 30 14:26:21.215575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.215533    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-vldc5"]
Oct 30 14:26:21.335448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.335418    4530 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 14:26:21.335618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.335478    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 14:26:21.878208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:21.878175    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:26:22.334899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:22.334860    4530 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 14:26:22.335374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:22.334922    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 14:26:23.118020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.117978    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61c8db3-2589-43d8-85a0-c49eccfe6023" path="/var/lib/kubelet/pods/d61c8db3-2589-43d8-85a0-c49eccfe6023/volumes"
Oct 30 14:26:23.118650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.118627    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" path="/var/lib/kubelet/pods/f4db2ee9-dd55-4d4a-89cc-f92c77d26db8/volumes"
Oct 30 14:26:23.175128 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.175097    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4" podUID="809d3ffe-ede2-4b4a-a8ff-38ba22659c28" containerName="main" containerID="cri-o://eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b" gracePeriod=45
Oct 30 14:26:23.335113 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.335082    4530 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 14:26:23.335452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.335136    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 14:26:23.513459 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.513432    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:26:23.535513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.535484    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-home\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-home\") pod \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") "
Oct 30 14:26:23.535513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.535517    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tmp-dir\") pod \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") "
Oct 30 14:26:23.535739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.535577    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tls-certs\" (UniqueName: \"kubernetes.io/secret/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tls-certs\") pod \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") "
Oct 30 14:26:23.535783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.535738    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-home" pod "809d3ffe-ede2-4b4a-a8ff-38ba22659c28" (UID: "809d3ffe-ede2-4b4a-a8ff-38ba22659c28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:23.535822 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.535779    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-model-cache\") pod \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") "
Oct 30 14:26:23.535871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.535846    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tmp-dir" pod "809d3ffe-ede2-4b4a-a8ff-38ba22659c28" (UID: "809d3ffe-ede2-4b4a-a8ff-38ba22659c28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:23.535943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.535910    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-kube-api-access-k6rvx\" (UniqueName: \"kubernetes.io/projected/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-kube-api-access-k6rvx\") pod \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") "
Oct 30 14:26:23.535989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.535957    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-dshm\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-dshm\") pod \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\" (UID: \"809d3ffe-ede2-4b4a-a8ff-38ba22659c28\") "
Oct 30 14:26:23.536027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.535989    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-model-cache" pod "809d3ffe-ede2-4b4a-a8ff-38ba22659c28" (UID: "809d3ffe-ede2-4b4a-a8ff-38ba22659c28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:23.536522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.536501    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:23.536522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.536520    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:23.536522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.536529    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:23.537990 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.537959    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tls-certs" pod "809d3ffe-ede2-4b4a-a8ff-38ba22659c28" (UID: "809d3ffe-ede2-4b4a-a8ff-38ba22659c28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:26:23.538088 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.538023    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-kube-api-access-k6rvx" pod "809d3ffe-ede2-4b4a-a8ff-38ba22659c28" (UID: "809d3ffe-ede2-4b4a-a8ff-38ba22659c28"). InnerVolumeSpecName "kube-api-access-k6rvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:23.538088 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.538027    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-dshm" pod "809d3ffe-ede2-4b4a-a8ff-38ba22659c28" (UID: "809d3ffe-ede2-4b4a-a8ff-38ba22659c28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:23.637686 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.637601    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6rvx\" (UniqueName: \"kubernetes.io/projected/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-kube-api-access-k6rvx\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:23.637686 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.637629    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:23.637686 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.637639    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809d3ffe-ede2-4b4a-a8ff-38ba22659c28-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:23.931728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:23.931644    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:26:24.047979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:24.047487    4530 generic.go:356] "Generic (PLEG): container finished" podID="809d3ffe-ede2-4b4a-a8ff-38ba22659c28" containerID="eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b" exitCode=0
Oct 30 14:26:24.047979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:24.047560    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"
Oct 30 14:26:24.047979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:24.047733    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4" event={"ID":"809d3ffe-ede2-4b4a-a8ff-38ba22659c28","Type":"ContainerDied","Data":"eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b"}
Oct 30 14:26:24.047979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:24.047773    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4" event={"ID":"809d3ffe-ede2-4b4a-a8ff-38ba22659c28","Type":"ContainerDied","Data":"c41df3bf9b30ec1959bc5ccf63711e4d7de4dddedb38f89c5b4df229a95b802f"}
Oct 30 14:26:24.047979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:24.047795    4530 scope.go:122] "RemoveContainer" containerID="eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b"
Oct 30 14:26:24.067473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:24.067447    4530 scope.go:122] "RemoveContainer" containerID="eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b"
Oct 30 14:26:24.067791 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:26:24.067758    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b\": container with ID starting with eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b not found: ID does not exist" containerID="eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b"
Oct 30 14:26:24.067890 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:24.067786    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b"} err="failed to get container status \"eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b\": rpc error: code = NotFound desc = could not find container \"eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b\": container with ID starting with eb9139f24b2df4bedec7722f5f6de936193a44d14c8fcc34e212f066853c2b3b not found: ID does not exist"
Oct 30 14:26:24.154750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:24.154717    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"]
Oct 30 14:26:24.160077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:24.160049    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4"]
Oct 30 14:26:24.334785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:24.334746    4530 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 14:26:24.334962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:24.334818    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 14:26:24.503974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:24.503933    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ca02c9/test-model-ca02c9-kserve-674db4d4dc-mbfz4" podUID="809d3ffe-ede2-4b4a-a8ff-38ba22659c28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:26:25.115856 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:25.115826    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809d3ffe-ede2-4b4a-a8ff-38ba22659c28" path="/var/lib/kubelet/pods/809d3ffe-ede2-4b4a-a8ff-38ba22659c28/volumes"
Oct 30 14:26:25.339646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:25.339618    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:26:25.340821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:25.340797    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:26:29.217000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:29.216820    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"]
Oct 30 14:26:29.217493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:29.217108    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97" podUID="736851b6-cadd-4b44-964c-57cd97372c82" containerName="maas-api" containerID="cri-o://76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38" gracePeriod=30
Oct 30 14:26:29.971706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:29.971683    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:26:30.048770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.048733    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-maas-api-tls\") pod \"736851b6-cadd-4b44-964c-57cd97372c82\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") "
Oct 30 14:26:30.048961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.048851    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736851b6-cadd-4b44-964c-57cd97372c82-kube-api-access-cdzk6\" (UniqueName: \"kubernetes.io/projected/736851b6-cadd-4b44-964c-57cd97372c82-kube-api-access-cdzk6\") pod \"736851b6-cadd-4b44-964c-57cd97372c82\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") "
Oct 30 14:26:30.049036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.048989    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736851b6-cadd-4b44-964c-57cd97372c82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/736851b6-cadd-4b44-964c-57cd97372c82-openshift-service-ca\") pod \"736851b6-cadd-4b44-964c-57cd97372c82\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") "
Oct 30 14:26:30.049036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.049025    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-metrics-certs\") pod \"736851b6-cadd-4b44-964c-57cd97372c82\" (UID: \"736851b6-cadd-4b44-964c-57cd97372c82\") "
Oct 30 14:26:30.049515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.049485    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736851b6-cadd-4b44-964c-57cd97372c82-openshift-service-ca" pod "736851b6-cadd-4b44-964c-57cd97372c82" (UID: "736851b6-cadd-4b44-964c-57cd97372c82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:26:30.050274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.050249    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/736851b6-cadd-4b44-964c-57cd97372c82-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:30.051302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.051275    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736851b6-cadd-4b44-964c-57cd97372c82-kube-api-access-cdzk6" pod "736851b6-cadd-4b44-964c-57cd97372c82" (UID: "736851b6-cadd-4b44-964c-57cd97372c82"). InnerVolumeSpecName "kube-api-access-cdzk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:30.051496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.051476    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-maas-api-tls" pod "736851b6-cadd-4b44-964c-57cd97372c82" (UID: "736851b6-cadd-4b44-964c-57cd97372c82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:26:30.052030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.052005    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-metrics-certs" pod "736851b6-cadd-4b44-964c-57cd97372c82" (UID: "736851b6-cadd-4b44-964c-57cd97372c82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:26:30.076727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.076699    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"]
Oct 30 14:26:30.077430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077414    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy"
Oct 30 14:26:30.077484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077432    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy"
Oct 30 14:26:30.077484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077441    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy"
Oct 30 14:26:30.077484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077446    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy"
Oct 30 14:26:30.077484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077460    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" containerName="istio-proxy"
Oct 30 14:26:30.077484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077466    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" containerName="istio-proxy"
Oct 30 14:26:30.077484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077478    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerName="istio-proxy"
Oct 30 14:26:30.077484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077483    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerName="istio-proxy"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077489    4530 cpu_manager.go:414] "containerMap: removing container" podUID="736851b6-cadd-4b44-964c-57cd97372c82" containerName="maas-api"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077495    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736851b6-cadd-4b44-964c-57cd97372c82" containerName="maas-api"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077505    4530 cpu_manager.go:414] "containerMap: removing container" podUID="809d3ffe-ede2-4b4a-a8ff-38ba22659c28" containerName="main"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077509    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809d3ffe-ede2-4b4a-a8ff-38ba22659c28" containerName="main"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077518    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d61c8db3-2589-43d8-85a0-c49eccfe6023" containerName="istio-proxy"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077522    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61c8db3-2589-43d8-85a0-c49eccfe6023" containerName="istio-proxy"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077656    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077663    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4db2ee9-dd55-4d4a-89cc-f92c77d26db8" containerName="istio-proxy"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077670    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6a1bff-3d6b-4fbb-85b4-3f61ad374dc0" containerName="istio-proxy"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077686    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8605fd0-853e-4cb6-9906-ceb6a07f8430" containerName="istio-proxy"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077693    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="736851b6-cadd-4b44-964c-57cd97372c82" containerName="maas-api"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077699    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="809d3ffe-ede2-4b4a-a8ff-38ba22659c28" containerName="main"
Oct 30 14:26:30.077751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.077706    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61c8db3-2589-43d8-85a0-c49eccfe6023" containerName="istio-proxy"
Oct 30 14:26:30.081444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.081426    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.102699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.102671    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b9c4b4-serving-cert\""
Oct 30 14:26:30.121390 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.121319    4530 generic.go:356] "Generic (PLEG): container finished" podID="736851b6-cadd-4b44-964c-57cd97372c82" containerID="76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38" exitCode=0
Oct 30 14:26:30.121580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.121557    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97" event={"ID":"736851b6-cadd-4b44-964c-57cd97372c82","Type":"ContainerDied","Data":"76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38"}
Oct 30 14:26:30.121675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.121625    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97" event={"ID":"736851b6-cadd-4b44-964c-57cd97372c82","Type":"ContainerDied","Data":"237f8296b901cdaa7cb52ab0bcec2d703652fef22d34c82f617f5de69f9fac58"}
Oct 30 14:26:30.121675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.121666    4530 scope.go:122] "RemoveContainer" containerID="76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38"
Oct 30 14:26:30.121795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.121765    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"
Oct 30 14:26:30.133012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.132979    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"]
Oct 30 14:26:30.146832 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.146807    4530 scope.go:122] "RemoveContainer" containerID="76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38"
Oct 30 14:26:30.147335 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:26:30.147309    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38\": container with ID starting with 76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38 not found: ID does not exist" containerID="76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38"
Oct 30 14:26:30.147495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.147468    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38"} err="failed to get container status \"76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38\": rpc error: code = NotFound desc = could not find container \"76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38\": container with ID starting with 76252f8d085b7a61cdf4d8022bc4c824c15ca160801c9de2bba7f05fbe7a3c38 not found: ID does not exist"
Oct 30 14:26:30.153607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.151674    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-metrics-certs\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.153607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.151715    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-maas-api-tls\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.153607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.151867    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1152d262-34c5-4c25-83d4-280abb53f12d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.153607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.151919    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m8k6\" (UniqueName: \"kubernetes.io/projected/1152d262-34c5-4c25-83d4-280abb53f12d-kube-api-access-5m8k6\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.153607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.152040    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdzk6\" (UniqueName: \"kubernetes.io/projected/736851b6-cadd-4b44-964c-57cd97372c82-kube-api-access-cdzk6\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:30.153607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.152050    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:30.153607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.152058    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/736851b6-cadd-4b44-964c-57cd97372c82-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:30.193434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.193403    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"]
Oct 30 14:26:30.197725 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.197702    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5a9d7b-86c5dd747-rgz97"]
Oct 30 14:26:30.253427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.253397    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-metrics-certs\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.253966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.253433    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-maas-api-tls\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.253966 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:26:30.253637    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-serving-cert: secret "maas-api-e2e-inf-b-b9c4b4-serving-cert" not found
Oct 30 14:26:30.253966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.253672    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1152d262-34c5-4c25-83d4-280abb53f12d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.253966 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:26:30.253714    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-maas-api-tls podName:1152d262-34c5-4c25-83d4-280abb53f12d nodeName:}" failed. No retries permitted until 2026-10-30 14:26:30.753692406 +0000 UTC m=+2255.785591330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-maas-api-tls") pod "maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt" (UID: "1152d262-34c5-4c25-83d4-280abb53f12d") : secret "maas-api-e2e-inf-b-b9c4b4-serving-cert" not found
Oct 30 14:26:30.253966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.253849    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m8k6\" (UniqueName: \"kubernetes.io/projected/1152d262-34c5-4c25-83d4-280abb53f12d-kube-api-access-5m8k6\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.254482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.254460    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1152d262-34c5-4c25-83d4-280abb53f12d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.256128 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.256108    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-metrics-certs\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.341486 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.341437    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m8k6\" (UniqueName: \"kubernetes.io/projected/1152d262-34c5-4c25-83d4-280abb53f12d-kube-api-access-5m8k6\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.762383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.762349    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-maas-api-tls\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:30.764973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:30.764946    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-maas-api-tls\") pod \"maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:31.021450 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:31.021382    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:31.163585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:31.161959    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736851b6-cadd-4b44-964c-57cd97372c82" path="/var/lib/kubelet/pods/736851b6-cadd-4b44-964c-57cd97372c82/volumes"
Oct 30 14:26:31.329888 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:26:31.329854    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1152d262_34c5_4c25_83d4_280abb53f12d.slice/crio-ce8f66ebd21ad90b235cf0f5b57c9f37f15a043eae64c6cc17c61426bbff703a WatchSource:0}: Error finding container ce8f66ebd21ad90b235cf0f5b57c9f37f15a043eae64c6cc17c61426bbff703a: Status 404 returned error can't find the container with id ce8f66ebd21ad90b235cf0f5b57c9f37f15a043eae64c6cc17c61426bbff703a
Oct 30 14:26:31.331972 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:31.331948    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"]
Oct 30 14:26:32.171315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:32.171278    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt" event={"ID":"1152d262-34c5-4c25-83d4-280abb53f12d","Type":"ContainerStarted","Data":"97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9"}
Oct 30 14:26:32.171522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:32.171332    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt" event={"ID":"1152d262-34c5-4c25-83d4-280abb53f12d","Type":"ContainerStarted","Data":"ce8f66ebd21ad90b235cf0f5b57c9f37f15a043eae64c6cc17c61426bbff703a"}
Oct 30 14:26:32.171522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:32.171352    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:32.224459 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:32.224410    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt" podStartSLOduration=1.9345391730000001 podStartE2EDuration="2.224393095s" totalImagesPullingTime="289.853922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:32.220847342 +0000 UTC m=+2257.252746283" watchObservedRunningTime="2026-10-30 14:26:32.224393095 +0000 UTC m=+2257.256292033"
Oct 30 14:26:38.195684 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:38.195648    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:26:38.358153 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:38.358119    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"]
Oct 30 14:26:38.358409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:38.358377    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy" containerID="cri-o://d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3" gracePeriod=30
Oct 30 14:26:41.367885 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:26:41.367845    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:26:43.647126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.647068    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:26:43.756170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.756117    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-envoy\") pod \"cdc89bfd-e9d1-4821-883c-18f959c7793f\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") "
Oct 30 14:26:43.756441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.756190    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc89bfd-e9d1-4821-883c-18f959c7793f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc89bfd-e9d1-4821-883c-18f959c7793f-istiod-ca-cert\") pod \"cdc89bfd-e9d1-4821-883c-18f959c7793f\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") "
Oct 30 14:26:43.756441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.756385    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-token\" (UniqueName: \"kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-token\") pod \"cdc89bfd-e9d1-4821-883c-18f959c7793f\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") "
Oct 30 14:26:43.756655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.756469    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-certs\") pod \"cdc89bfd-e9d1-4821-883c-18f959c7793f\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") "
Oct 30 14:26:43.756655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.756531    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-credential-socket\") pod \"cdc89bfd-e9d1-4821-883c-18f959c7793f\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") "
Oct 30 14:26:43.756655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.756623    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-data\") pod \"cdc89bfd-e9d1-4821-883c-18f959c7793f\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") "
Oct 30 14:26:43.757331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.756653    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-podinfo\") pod \"cdc89bfd-e9d1-4821-883c-18f959c7793f\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") "
Oct 30 14:26:43.757331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.756716    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-kube-api-access-5z97m\" (UniqueName: \"kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-kube-api-access-5z97m\") pod \"cdc89bfd-e9d1-4821-883c-18f959c7793f\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") "
Oct 30 14:26:43.757331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.756757    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-socket\") pod \"cdc89bfd-e9d1-4821-883c-18f959c7793f\" (UID: \"cdc89bfd-e9d1-4821-883c-18f959c7793f\") "
Oct 30 14:26:43.758585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.758461    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-data" pod "cdc89bfd-e9d1-4821-883c-18f959c7793f" (UID: "cdc89bfd-e9d1-4821-883c-18f959c7793f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:43.759015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.758989    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-certs" pod "cdc89bfd-e9d1-4821-883c-18f959c7793f" (UID: "cdc89bfd-e9d1-4821-883c-18f959c7793f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:43.759759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.759182    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-socket" pod "cdc89bfd-e9d1-4821-883c-18f959c7793f" (UID: "cdc89bfd-e9d1-4821-883c-18f959c7793f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:43.759759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.759355    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-credential-socket" pod "cdc89bfd-e9d1-4821-883c-18f959c7793f" (UID: "cdc89bfd-e9d1-4821-883c-18f959c7793f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:43.760279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.760256    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc89bfd-e9d1-4821-883c-18f959c7793f-istiod-ca-cert" pod "cdc89bfd-e9d1-4821-883c-18f959c7793f" (UID: "cdc89bfd-e9d1-4821-883c-18f959c7793f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:26:43.775755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.762094    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-podinfo" pod "cdc89bfd-e9d1-4821-883c-18f959c7793f" (UID: "cdc89bfd-e9d1-4821-883c-18f959c7793f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:26:43.775755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.774751    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-kube-api-access-5z97m" pod "cdc89bfd-e9d1-4821-883c-18f959c7793f" (UID: "cdc89bfd-e9d1-4821-883c-18f959c7793f"). InnerVolumeSpecName "kube-api-access-5z97m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:43.775755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.775670    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-token" pod "cdc89bfd-e9d1-4821-883c-18f959c7793f" (UID: "cdc89bfd-e9d1-4821-883c-18f959c7793f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:26:43.779524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.779498    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-envoy" pod "cdc89bfd-e9d1-4821-883c-18f959c7793f" (UID: "cdc89bfd-e9d1-4821-883c-18f959c7793f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:26:43.859116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.859081    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:43.859116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.859112    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:43.859297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.859126    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:43.859297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.859139    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:43.859297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.859152    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:43.859297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.859163    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z97m\" (UniqueName: \"kubernetes.io/projected/cdc89bfd-e9d1-4821-883c-18f959c7793f-kube-api-access-5z97m\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:43.859297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.859177    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:43.859297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.859189    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc89bfd-e9d1-4821-883c-18f959c7793f-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:43.859297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:43.859201    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc89bfd-e9d1-4821-883c-18f959c7793f-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:26:44.284180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.284149    4530 generic.go:356] "Generic (PLEG): container finished" podID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerID="d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3" exitCode=0
Oct 30 14:26:44.284383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.284201    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"
Oct 30 14:26:44.284383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.284233    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" event={"ID":"cdc89bfd-e9d1-4821-883c-18f959c7793f","Type":"ContainerDied","Data":"d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3"}
Oct 30 14:26:44.284383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.284301    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t" event={"ID":"cdc89bfd-e9d1-4821-883c-18f959c7793f","Type":"ContainerDied","Data":"7249e92d9ab31735131385b775a63238d623884795e3cfd2c8ebabeb8a49d2ab"}
Oct 30 14:26:44.284383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.284326    4530 scope.go:122] "RemoveContainer" containerID="d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3"
Oct 30 14:26:44.296142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.296126    4530 scope.go:122] "RemoveContainer" containerID="029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9"
Oct 30 14:26:44.305996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.305979    4530 scope.go:122] "RemoveContainer" containerID="d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3"
Oct 30 14:26:44.306235 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:26:44.306216    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3\": container with ID starting with d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3 not found: ID does not exist" containerID="d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3"
Oct 30 14:26:44.306299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.306243    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3"} err="failed to get container status \"d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3\": rpc error: code = NotFound desc = could not find container \"d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3\": container with ID starting with d42b11931c7755a67726faf0c8a26671636fa4dbab2da2992d8b9de2e30caef3 not found: ID does not exist"
Oct 30 14:26:44.306299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.306260    4530 scope.go:122] "RemoveContainer" containerID="029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9"
Oct 30 14:26:44.306483 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:26:44.306462    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9\": container with ID starting with 029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9 not found: ID does not exist" containerID="029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9"
Oct 30 14:26:44.306531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.306491    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9"} err="failed to get container status \"029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9\": rpc error: code = NotFound desc = could not find container \"029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9\": container with ID starting with 029c03bc6bb305561e43303cfb374507009f521c2d880c6d05bb86495f0748f9 not found: ID does not exist"
Oct 30 14:26:44.367593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.367560    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"]
Oct 30 14:26:44.381490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.381463    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5a9d7b-openshift-default-796bc9cccf-z2m7t"]
Oct 30 14:26:44.756086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.756052    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"]
Oct 30 14:26:44.756998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.756975    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy"
Oct 30 14:26:44.757129 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.756998    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy"
Oct 30 14:26:44.757129 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.757027    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy"
Oct 30 14:26:44.757129 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.757036    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy"
Oct 30 14:26:44.757289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.757228    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy"
Oct 30 14:26:44.757289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.757257    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" containerName="istio-proxy"
Oct 30 14:26:44.766951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.766927    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.779526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.779503    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"]
Oct 30 14:26:44.785478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.785369    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d15602\"/\"kube-root-ca.crt\""
Oct 30 14:26:44.785478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.785447    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d15602\"/\"openshift-service-ca.crt\""
Oct 30 14:26:44.785741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.785651    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d15602\"/\"default-dockercfg-bng57\""
Oct 30 14:26:44.785741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.785719    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d15602\"/\"test-model-d15602-kserve-self-signed-certs\""
Oct 30 14:26:44.875073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.875044    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-tmp-dir\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.875225 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.875109    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-home\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.875267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.875226    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-model-cache\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.875340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.875325    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qr96\" (UniqueName: \"kubernetes.io/projected/8740a5a1-f59a-4de3-aa24-0e235518ebed-kube-api-access-4qr96\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.875378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.875360    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8740a5a1-f59a-4de3-aa24-0e235518ebed-tls-certs\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.875508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.875490    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-dshm\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.977219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.977186    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-dshm\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.977394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.977247    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-tmp-dir\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.977394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.977318    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-home\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.977582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.977393    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-model-cache\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.977582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.977494    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qr96\" (UniqueName: \"kubernetes.io/projected/8740a5a1-f59a-4de3-aa24-0e235518ebed-kube-api-access-4qr96\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.977582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.977575    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8740a5a1-f59a-4de3-aa24-0e235518ebed-tls-certs\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.977794 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.977719    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-tmp-dir\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.977794 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.977779    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-model-cache\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.977909 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.977793    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-home\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.979633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.979608    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-dshm\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:44.980182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:44.980161    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8740a5a1-f59a-4de3-aa24-0e235518ebed-tls-certs\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:45.004343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:45.004320    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qr96\" (UniqueName: \"kubernetes.io/projected/8740a5a1-f59a-4de3-aa24-0e235518ebed-kube-api-access-4qr96\") pod \"test-model-d15602-kserve-c9cfcd95f-w9t7r\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") " pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:45.094562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:45.094059    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:45.174901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:45.174848    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc89bfd-e9d1-4821-883c-18f959c7793f" path="/var/lib/kubelet/pods/cdc89bfd-e9d1-4821-883c-18f959c7793f/volumes"
Oct 30 14:26:45.522418 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:26:45.522391    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8740a5a1_f59a_4de3_aa24_0e235518ebed.slice/crio-e0815b130cf860a662c88fffdd5583bf5e69d4cb7ef9b99aef592e33f26fd6b9 WatchSource:0}: Error finding container e0815b130cf860a662c88fffdd5583bf5e69d4cb7ef9b99aef592e33f26fd6b9: Status 404 returned error can't find the container with id e0815b130cf860a662c88fffdd5583bf5e69d4cb7ef9b99aef592e33f26fd6b9
Oct 30 14:26:45.525264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:45.525212    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"]
Oct 30 14:26:46.319826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:46.319791    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r" event={"ID":"8740a5a1-f59a-4de3-aa24-0e235518ebed","Type":"ContainerStarted","Data":"b9be0d11ae5fd5fa04e567f892e42a4eb082f2fce95ae71ec9a14214488607da"}
Oct 30 14:26:46.320290 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:46.319837    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r" event={"ID":"8740a5a1-f59a-4de3-aa24-0e235518ebed","Type":"ContainerStarted","Data":"e0815b130cf860a662c88fffdd5583bf5e69d4cb7ef9b99aef592e33f26fd6b9"}
Oct 30 14:26:46.369191 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:46.369129    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r" podStartSLOduration=2.369115615 podStartE2EDuration="2.369115615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:46.36700535 +0000 UTC m=+2271.398904290" watchObservedRunningTime="2026-10-30 14:26:46.369115615 +0000 UTC m=+2271.401014554"
Oct 30 14:26:47.096709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:47.096667    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:49.351563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:49.351491    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:26:55.102565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:55.101993    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:55.121999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:55.121971    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:55.459935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:55.459859    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:26:58.243223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:58.243186    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"]
Oct 30 14:26:58.243663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:26:58.243374    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" podUID="cb30f186-609d-4e05-9018-9ce4b63a133c" containerName="istio-proxy" containerID="cri-o://b11fd375802da1ba34ab1ec0f63f91e44d145e2a6c0db8ec0409b6b122581a12" gracePeriod=30
Oct 30 14:27:01.508328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:01.508296    4530 generic.go:356] "Generic (PLEG): container finished" podID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerID="5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a" exitCode=0
Oct 30 14:27:01.508847 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:01.508379    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" event={"ID":"e271438d-7e48-47fe-887b-d3ba07d2a875","Type":"ContainerDied","Data":"5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a"}
Oct 30 14:27:01.508893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:01.508862    4530 scope.go:122] "RemoveContainer" containerID="5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a"
Oct 30 14:27:01.510766 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:01.510735    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:27:01.510850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:01.510809    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:27:01.510850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:01.510844    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:27:02.181586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:02.181527    4530 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:27:02.181697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:02.181612    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" podUID="cb30f186-609d-4e05-9018-9ce4b63a133c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:27:02.517967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:02.517880    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" event={"ID":"e271438d-7e48-47fe-887b-d3ba07d2a875","Type":"ContainerStarted","Data":"076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105"}
Oct 30 14:27:02.818213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:02.818176    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:27:02.818392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:02.818372    4530 patch_prober.go:29] interesting pod/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 14:27:02.818472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:02.818415    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 14:27:03.527609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.527574    4530 generic.go:356] "Generic (PLEG): container finished" podID="cb30f186-609d-4e05-9018-9ce4b63a133c" containerID="b11fd375802da1ba34ab1ec0f63f91e44d145e2a6c0db8ec0409b6b122581a12" exitCode=0
Oct 30 14:27:03.528090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.527651    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" event={"ID":"cb30f186-609d-4e05-9018-9ce4b63a133c","Type":"ContainerDied","Data":"b11fd375802da1ba34ab1ec0f63f91e44d145e2a6c0db8ec0409b6b122581a12"}
Oct 30 14:27:03.605622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.605585    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:27:03.664201 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.664169    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-kube-api-access-72smx\" (UniqueName: \"kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-kube-api-access-72smx\") pod \"cb30f186-609d-4e05-9018-9ce4b63a133c\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") "
Oct 30 14:27:03.664379 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.664283    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-token\" (UniqueName: \"kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-token\") pod \"cb30f186-609d-4e05-9018-9ce4b63a133c\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") "
Oct 30 14:27:03.664452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.664404    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-podinfo\") pod \"cb30f186-609d-4e05-9018-9ce4b63a133c\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") "
Oct 30 14:27:03.664525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.664509    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-socket\") pod \"cb30f186-609d-4e05-9018-9ce4b63a133c\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") "
Oct 30 14:27:03.664632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.664617    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-certs\") pod \"cb30f186-609d-4e05-9018-9ce4b63a133c\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") "
Oct 30 14:27:03.664693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.664671    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-envoy\") pod \"cb30f186-609d-4e05-9018-9ce4b63a133c\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") "
Oct 30 14:27:03.664783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.664760    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-credential-socket\") pod \"cb30f186-609d-4e05-9018-9ce4b63a133c\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") "
Oct 30 14:27:03.664850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.664784    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-socket" pod "cb30f186-609d-4e05-9018-9ce4b63a133c" (UID: "cb30f186-609d-4e05-9018-9ce4b63a133c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:27:03.664850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.664811    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-data\") pod \"cb30f186-609d-4e05-9018-9ce4b63a133c\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") "
Oct 30 14:27:03.664967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.664848    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb30f186-609d-4e05-9018-9ce4b63a133c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb30f186-609d-4e05-9018-9ce4b63a133c-istiod-ca-cert\") pod \"cb30f186-609d-4e05-9018-9ce4b63a133c\" (UID: \"cb30f186-609d-4e05-9018-9ce4b63a133c\") "
Oct 30 14:27:03.664967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.664895    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-certs" pod "cb30f186-609d-4e05-9018-9ce4b63a133c" (UID: "cb30f186-609d-4e05-9018-9ce4b63a133c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:27:03.665151 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.665127    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-data" pod "cb30f186-609d-4e05-9018-9ce4b63a133c" (UID: "cb30f186-609d-4e05-9018-9ce4b63a133c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:27:03.665408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.665383    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb30f186-609d-4e05-9018-9ce4b63a133c-istiod-ca-cert" pod "cb30f186-609d-4e05-9018-9ce4b63a133c" (UID: "cb30f186-609d-4e05-9018-9ce4b63a133c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:27:03.665728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.665706    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:03.665812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.665732    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb30f186-609d-4e05-9018-9ce4b63a133c-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:03.665812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.665746    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:03.665812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.665759    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:03.665916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.665051    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-credential-socket" pod "cb30f186-609d-4e05-9018-9ce4b63a133c" (UID: "cb30f186-609d-4e05-9018-9ce4b63a133c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:27:03.666741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.666712    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-kube-api-access-72smx" pod "cb30f186-609d-4e05-9018-9ce4b63a133c" (UID: "cb30f186-609d-4e05-9018-9ce4b63a133c"). InnerVolumeSpecName "kube-api-access-72smx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:27:03.666851 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.666749    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-token" pod "cb30f186-609d-4e05-9018-9ce4b63a133c" (UID: "cb30f186-609d-4e05-9018-9ce4b63a133c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:27:03.666964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.666943    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-podinfo" pod "cb30f186-609d-4e05-9018-9ce4b63a133c" (UID: "cb30f186-609d-4e05-9018-9ce4b63a133c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:27:03.667090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.667072    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-envoy" pod "cb30f186-609d-4e05-9018-9ce4b63a133c" (UID: "cb30f186-609d-4e05-9018-9ce4b63a133c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:27:03.766926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.766889    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:03.766926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.766919    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72smx\" (UniqueName: \"kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-kube-api-access-72smx\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:03.767122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.766935    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:03.767122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.766945    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:03.767122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.766953    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb30f186-609d-4e05-9018-9ce4b63a133c-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:03.818570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.818470    4530 patch_prober.go:29] interesting pod/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 14:27:03.818570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:03.818533    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 14:27:04.537230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:04.537192    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h" event={"ID":"cb30f186-609d-4e05-9018-9ce4b63a133c","Type":"ContainerDied","Data":"3f0d9591ee75d304709616c2ae0d96aa6c58b9c288ef8cf68f069bbba50d3342"}
Oct 30 14:27:04.537230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:04.537217    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"
Oct 30 14:27:04.537712 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:04.537261    4530 scope.go:122] "RemoveContainer" containerID="b11fd375802da1ba34ab1ec0f63f91e44d145e2a6c0db8ec0409b6b122581a12"
Oct 30 14:27:04.624391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:04.624358    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"]
Oct 30 14:27:04.628768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:04.628741    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-vg97h"]
Oct 30 14:27:04.841571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:04.818875    4530 patch_prober.go:29] interesting pod/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 14:27:04.841571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:04.818943    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 14:27:05.155620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:05.154351    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb30f186-609d-4e05-9018-9ce4b63a133c" path="/var/lib/kubelet/pods/cb30f186-609d-4e05-9018-9ce4b63a133c/volumes"
Oct 30 14:27:05.818298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:05.818261    4530 patch_prober.go:29] interesting pod/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 14:27:05.818814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:05.818321    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 14:27:06.818642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:06.818606    4530 patch_prober.go:29] interesting pod/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 14:27:06.819049 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:06.818667    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 14:27:07.818240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:07.818160    4530 patch_prober.go:29] interesting pod/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 14:27:07.818383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:07.818234    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 14:27:08.823204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:08.823167    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:27:08.823724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:08.823641    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:27:08.824224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:08.824204    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:27:11.419129 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:27:11.419089    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:27:12.315421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.315382    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"]
Oct 30 14:27:12.316793 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.316761    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cb30f186-609d-4e05-9018-9ce4b63a133c" containerName="istio-proxy"
Oct 30 14:27:12.316793 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.316792    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb30f186-609d-4e05-9018-9ce4b63a133c" containerName="istio-proxy"
Oct 30 14:27:12.317016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.317001    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb30f186-609d-4e05-9018-9ce4b63a133c" containerName="istio-proxy"
Oct 30 14:27:12.321772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.321745    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.338698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.338668    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"]
Oct 30 14:27:12.346094 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.346070    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-93549e\"/\"openshift-service-ca.crt\""
Oct 30 14:27:12.346278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.346250    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-93549e\"/\"kube-root-ca.crt\""
Oct 30 14:27:12.353254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.353235    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-93549e\"/\"test-model-93549e-kserve-self-signed-certs\""
Oct 30 14:27:12.353254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.353234    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-93549e\"/\"default-dockercfg-fhlt7\""
Oct 30 14:27:12.372233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.372210    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-dshm\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.372364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.372263    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jzn\" (UniqueName: \"kubernetes.io/projected/6be6f5b8-5567-4c76-96be-dcdc904c4f37-kube-api-access-v2jzn\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.372443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.372396    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-home\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.372506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.372489    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tls-certs\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.372672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.372518    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tmp-dir\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.372672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.372605    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-model-cache\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.474071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.474032    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jzn\" (UniqueName: \"kubernetes.io/projected/6be6f5b8-5567-4c76-96be-dcdc904c4f37-kube-api-access-v2jzn\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.474422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.474118    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-home\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.474422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.474190    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tls-certs\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.474422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.474229    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tmp-dir\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.474422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.474263    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-model-cache\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.474682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.474418    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-dshm\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.474682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.474590    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-home\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.474682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.474652    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tmp-dir\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.474828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.474739    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-model-cache\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.476592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.476563    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-dshm\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.476838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.476821    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tls-certs\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.500595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.500571    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jzn\" (UniqueName: \"kubernetes.io/projected/6be6f5b8-5567-4c76-96be-dcdc904c4f37-kube-api-access-v2jzn\") pod \"test-model-93549e-kserve-7c769d45f7-j6s5m\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") " pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.643458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.643282    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:12.966856 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:27:12.966829    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6be6f5b8_5567_4c76_96be_dcdc904c4f37.slice/crio-840ddf1205b362e29c8ebae57992839563043ad2c115d0cff5c46204bdeecd33 WatchSource:0}: Error finding container 840ddf1205b362e29c8ebae57992839563043ad2c115d0cff5c46204bdeecd33: Status 404 returned error can't find the container with id 840ddf1205b362e29c8ebae57992839563043ad2c115d0cff5c46204bdeecd33
Oct 30 14:27:12.968510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:12.968402    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"]
Oct 30 14:27:13.652263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:13.652230    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m" event={"ID":"6be6f5b8-5567-4c76-96be-dcdc904c4f37","Type":"ContainerStarted","Data":"8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e"}
Oct 30 14:27:13.652714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:13.652278    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m" event={"ID":"6be6f5b8-5567-4c76-96be-dcdc904c4f37","Type":"ContainerStarted","Data":"840ddf1205b362e29c8ebae57992839563043ad2c115d0cff5c46204bdeecd33"}
Oct 30 14:27:13.703363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:13.703305    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m" podStartSLOduration=1.703291219 podStartE2EDuration="1.703291219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:13.699058484 +0000 UTC m=+2298.730957425" watchObservedRunningTime="2026-10-30 14:27:13.703291219 +0000 UTC m=+2298.735190173"
Oct 30 14:27:14.644624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:14.644593    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:17.534408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.534376    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"]
Oct 30 14:27:17.538714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.538683    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.561368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.561341    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"]
Oct 30 14:27:17.568819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.568788    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f3c580\"/\"kube-root-ca.crt\""
Oct 30 14:27:17.568819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.568788    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f3c580\"/\"openshift-service-ca.crt\""
Oct 30 14:27:17.569880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.569850    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f3c580\"/\"default-dockercfg-x6n6k\""
Oct 30 14:27:17.569880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.569852    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f3c580\"/\"test-model-f3c580-kserve-self-signed-certs\""
Oct 30 14:27:17.661722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.661689    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-home\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.661722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.661722    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-model-cache\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.661918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.661859    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-dshm\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.661918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.661883    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-tmp-dir\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.661991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.661970    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c79s\" (UniqueName: \"kubernetes.io/projected/e02d6098-1f27-4b24-b0bc-49ff80d17724-kube-api-access-9c79s\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.662102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.662086    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02d6098-1f27-4b24-b0bc-49ff80d17724-tls-certs\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.765991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.765516    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-home\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.778672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.778634    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-model-cache\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.779498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.779472    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-dshm\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.779720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.779699    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-tmp-dir\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.779924 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.779905    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c79s\" (UniqueName: \"kubernetes.io/projected/e02d6098-1f27-4b24-b0bc-49ff80d17724-kube-api-access-9c79s\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.780152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.780135    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02d6098-1f27-4b24-b0bc-49ff80d17724-tls-certs\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.798254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.765950    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-home\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.808449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.779252    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-model-cache\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.808449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.782174    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-tmp-dir\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.808449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.801305    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02d6098-1f27-4b24-b0bc-49ff80d17724-tls-certs\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.822276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.819447    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-dshm\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:17.879571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:17.856934    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c79s\" (UniqueName: \"kubernetes.io/projected/e02d6098-1f27-4b24-b0bc-49ff80d17724-kube-api-access-9c79s\") pod \"test-model-f3c580-kserve-54f79f67cc-rwbm2\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") " pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:18.152065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.151984    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:18.456689 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:27:18.456659    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode02d6098_1f27_4b24_b0bc_49ff80d17724.slice/crio-13e737ddb72996d06b10e50269fea15fdda795d2c70ef7ec6b9640ea5508e576 WatchSource:0}: Error finding container 13e737ddb72996d06b10e50269fea15fdda795d2c70ef7ec6b9640ea5508e576: Status 404 returned error can't find the container with id 13e737ddb72996d06b10e50269fea15fdda795d2c70ef7ec6b9640ea5508e576
Oct 30 14:27:18.459932 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.459905    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"]
Oct 30 14:27:18.695755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.695714    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2" event={"ID":"e02d6098-1f27-4b24-b0bc-49ff80d17724","Type":"ContainerStarted","Data":"4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a"}
Oct 30 14:27:18.695755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.695763    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2" event={"ID":"e02d6098-1f27-4b24-b0bc-49ff80d17724","Type":"ContainerStarted","Data":"13e737ddb72996d06b10e50269fea15fdda795d2c70ef7ec6b9640ea5508e576"}
Oct 30 14:27:18.746676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.746596    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"]
Oct 30 14:27:18.747211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.747171    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2" podStartSLOduration=1.747158608 podStartE2EDuration="1.747158608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:18.744817222 +0000 UTC m=+2303.776716162" watchObservedRunningTime="2026-10-30 14:27:18.747158608 +0000 UTC m=+2303.779057547"
Oct 30 14:27:18.750081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.750063    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:18.791254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.791220    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"]
Oct 30 14:27:18.797834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.797788    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-019e5b-serving-cert\""
Oct 30 14:27:18.907308 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.907280    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-metrics-certs\") pod \"maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:18.907508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.907350    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mngfp\" (UniqueName: \"kubernetes.io/projected/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-kube-api-access-mngfp\") pod \"maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:18.907508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.907435    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:18.907508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:18.907471    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:19.009439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:19.009325    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-metrics-certs\") pod \"maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:19.009657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:19.009508    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mngfp\" (UniqueName: \"kubernetes.io/projected/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-kube-api-access-mngfp\") pod \"maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:19.009657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:19.009552    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:19.009657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:19.009580    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:19.012677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:19.012644    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:19.016476 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:19.016427    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:19.016637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:19.016589    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-metrics-certs\") pod \"maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:19.036622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:19.036597    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mngfp\" (UniqueName: \"kubernetes.io/projected/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-kube-api-access-mngfp\") pod \"maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:19.064777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:19.064753    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:19.152722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:19.152689    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:19.369759 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:27:19.369728    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aeffd3a_6f61_4cca_8a77_0b2a337639f2.slice/crio-2f3894fe316a7f9344e473f945ffe3086103e17c04397cff76c92d09ccf1b791 WatchSource:0}: Error finding container 2f3894fe316a7f9344e473f945ffe3086103e17c04397cff76c92d09ccf1b791: Status 404 returned error can't find the container with id 2f3894fe316a7f9344e473f945ffe3086103e17c04397cff76c92d09ccf1b791
Oct 30 14:27:19.372861 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:19.372834    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"]
Oct 30 14:27:19.705205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:19.705130    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq" event={"ID":"7aeffd3a-6f61-4cca-8a77-0b2a337639f2","Type":"ContainerStarted","Data":"2f3894fe316a7f9344e473f945ffe3086103e17c04397cff76c92d09ccf1b791"}
Oct 30 14:27:20.714816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:20.714775    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq" event={"ID":"7aeffd3a-6f61-4cca-8a77-0b2a337639f2","Type":"ContainerStarted","Data":"5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c"}
Oct 30 14:27:20.715212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:20.714846    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:20.766164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:20.766112    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq" podStartSLOduration=2.413426126 podStartE2EDuration="2.76609937s" totalImagesPullingTime="352.673244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:20.764132478 +0000 UTC m=+2305.796031417" watchObservedRunningTime="2026-10-30 14:27:20.76609937 +0000 UTC m=+2305.797998356"
Oct 30 14:27:22.644733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:22.644696    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:22.657494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:22.657458    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:22.742061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:22.742027    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:27:26.729466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:26.729435    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:27:28.153089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:28.153048    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:28.165672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:28.165649    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:28.838901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:28.838867    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:27:30.210583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.210525    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"]
Oct 30 14:27:30.211037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.210784    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm" podUID="2eb051b8-57d4-430e-b039-6abb961d0fb9" containerName="maas-api" containerID="cri-o://10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c" gracePeriod=30
Oct 30 14:27:30.610588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.610562    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:27:30.763043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.763001    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb051b8-57d4-430e-b039-6abb961d0fb9-kube-api-access-8chgh\" (UniqueName: \"kubernetes.io/projected/2eb051b8-57d4-430e-b039-6abb961d0fb9-kube-api-access-8chgh\") pod \"2eb051b8-57d4-430e-b039-6abb961d0fb9\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") "
Oct 30 14:27:30.763952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.763920    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-maas-api-tls\") pod \"2eb051b8-57d4-430e-b039-6abb961d0fb9\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") "
Oct 30 14:27:30.764255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.764234    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-metrics-certs\") pod \"2eb051b8-57d4-430e-b039-6abb961d0fb9\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") "
Oct 30 14:27:30.764476 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.764457    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb051b8-57d4-430e-b039-6abb961d0fb9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb051b8-57d4-430e-b039-6abb961d0fb9-openshift-service-ca\") pod \"2eb051b8-57d4-430e-b039-6abb961d0fb9\" (UID: \"2eb051b8-57d4-430e-b039-6abb961d0fb9\") "
Oct 30 14:27:30.780621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.773337    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb051b8-57d4-430e-b039-6abb961d0fb9-kube-api-access-8chgh" pod "2eb051b8-57d4-430e-b039-6abb961d0fb9" (UID: "2eb051b8-57d4-430e-b039-6abb961d0fb9"). InnerVolumeSpecName "kube-api-access-8chgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:27:30.785283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.785056    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-metrics-certs" pod "2eb051b8-57d4-430e-b039-6abb961d0fb9" (UID: "2eb051b8-57d4-430e-b039-6abb961d0fb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:27:30.786139 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.785685    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb051b8-57d4-430e-b039-6abb961d0fb9-openshift-service-ca" pod "2eb051b8-57d4-430e-b039-6abb961d0fb9" (UID: "2eb051b8-57d4-430e-b039-6abb961d0fb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:27:30.809766 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.808864    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-maas-api-tls" pod "2eb051b8-57d4-430e-b039-6abb961d0fb9" (UID: "2eb051b8-57d4-430e-b039-6abb961d0fb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:27:30.841808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.832741    4530 generic.go:356] "Generic (PLEG): container finished" podID="2eb051b8-57d4-430e-b039-6abb961d0fb9" containerID="10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c" exitCode=0
Oct 30 14:27:30.841808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.832896    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm" event={"ID":"2eb051b8-57d4-430e-b039-6abb961d0fb9","Type":"ContainerDied","Data":"10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c"}
Oct 30 14:27:30.841808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.832956    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm" event={"ID":"2eb051b8-57d4-430e-b039-6abb961d0fb9","Type":"ContainerDied","Data":"57e24917b820e8df0fb8c948f16f0e297245367863e4ff13e5f529e0f0c826b6"}
Oct 30 14:27:30.841808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.832985    4530 scope.go:122] "RemoveContainer" containerID="10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c"
Oct 30 14:27:30.841808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.833161    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"
Oct 30 14:27:30.851001 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.850972    4530 scope.go:122] "RemoveContainer" containerID="10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c"
Oct 30 14:27:30.852606 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:27:30.852534    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c\": container with ID starting with 10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c not found: ID does not exist" containerID="10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c"
Oct 30 14:27:30.852753 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.852614    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c"} err="failed to get container status \"10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c\": rpc error: code = NotFound desc = could not find container \"10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c\": container with ID starting with 10427b56372f51715fb50cee88f448d5ba52fc60be4eb68bdfcaeb6f3d30903c not found: ID does not exist"
Oct 30 14:27:30.877865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.877840    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:30.877865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.877863    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb051b8-57d4-430e-b039-6abb961d0fb9-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:30.878046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.877874    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb051b8-57d4-430e-b039-6abb961d0fb9-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:30.878046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.877884    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8chgh\" (UniqueName: \"kubernetes.io/projected/2eb051b8-57d4-430e-b039-6abb961d0fb9-kube-api-access-8chgh\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:27:30.899312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.899282    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"]
Oct 30 14:27:30.911974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:30.911940    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ca02c9-95b4dfd56-87qdm"]
Oct 30 14:27:31.117635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:31.117531    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb051b8-57d4-430e-b039-6abb961d0fb9" path="/var/lib/kubelet/pods/2eb051b8-57d4-430e-b039-6abb961d0fb9/volumes"
Oct 30 14:27:32.133724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:32.133683    4530 patch_prober.go:29] interesting pod/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:27:32.134226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:32.133751    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:27:33.863000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:33.862970    4530 generic.go:356] "Generic (PLEG): container finished" podID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerID="e7460e84dd32a353e5f78cbaa9bb41be362e6514c17d918359dea4465abc2dfc" exitCode=0
Oct 30 14:27:33.863457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:33.863042    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" event={"ID":"06147d62-09b3-4e0e-bb3a-a8fad254a8dd","Type":"ContainerDied","Data":"e7460e84dd32a353e5f78cbaa9bb41be362e6514c17d918359dea4465abc2dfc"}
Oct 30 14:27:33.863517 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:33.863462    4530 scope.go:122] "RemoveContainer" containerID="e7460e84dd32a353e5f78cbaa9bb41be362e6514c17d918359dea4465abc2dfc"
Oct 30 14:27:33.865173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:33.865145    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:27:33.865235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:33.865212    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:27:33.865272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:33.865247    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:27:34.873745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:34.873709    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" event={"ID":"06147d62-09b3-4e0e-bb3a-a8fad254a8dd","Type":"ContainerStarted","Data":"e1fd18b51491b08c1d1c5f5e0ff01aa752623fd7f1330ed80fc15d483695a31a"}
Oct 30 14:27:35.808567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:35.806996    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:27:35.808567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:35.807113    4530 patch_prober.go:29] interesting pod/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 14:27:35.808567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:35.807156    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 14:27:36.537367 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:36.537329    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:27:36.806210 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:36.806167    4530 patch_prober.go:29] interesting pod/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 14:27:36.806406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:36.806228    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 14:27:37.815565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:37.810910    4530 patch_prober.go:29] interesting pod/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 14:27:37.815565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:37.811008    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 14:27:38.807665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:38.807626    4530 patch_prober.go:29] interesting pod/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 14:27:38.807980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:38.807948    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 14:27:39.807225 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.807183    4530 patch_prober.go:29] interesting pod/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 14:27:39.807784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.807253    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 14:27:39.937195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.937158    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"]
Oct 30 14:27:39.938434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.938395    4530 cpu_manager.go:414] "containerMap: removing container" podUID="2eb051b8-57d4-430e-b039-6abb961d0fb9" containerName="maas-api"
Oct 30 14:27:39.938434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.938424    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb051b8-57d4-430e-b039-6abb961d0fb9" containerName="maas-api"
Oct 30 14:27:39.938672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.938556    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb051b8-57d4-430e-b039-6abb961d0fb9" containerName="maas-api"
Oct 30 14:27:39.942010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.941985    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:39.965249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.965214    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"]
Oct 30 14:27:39.968982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.968958    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0b1c15\"/\"test-model-0b1c15-kserve-self-signed-certs\""
Oct 30 14:27:39.969117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.969032    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0b1c15\"/\"default-dockercfg-mcbzh\""
Oct 30 14:27:39.973915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.973856    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0b1c15\"/\"openshift-service-ca.crt\""
Oct 30 14:27:39.974110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.973939    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0b1c15\"/\"kube-root-ca.crt\""
Oct 30 14:27:39.977781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.977759    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-dshm\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:39.977943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.977821    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9wbw\" (UniqueName: \"kubernetes.io/projected/4c33c19b-0057-40dd-af41-fe69990fb374-kube-api-access-k9wbw\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:39.977943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.977922    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-home\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:39.978136 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.978009    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-tmp-dir\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:39.978136 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.978129    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-model-cache\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:39.978260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:39.978157    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c33c19b-0057-40dd-af41-fe69990fb374-tls-certs\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.094609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.080079    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9wbw\" (UniqueName: \"kubernetes.io/projected/4c33c19b-0057-40dd-af41-fe69990fb374-kube-api-access-k9wbw\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.094609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.080198    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-home\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.094609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.080286    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-tmp-dir\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.094609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.080404    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-model-cache\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.094609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.080429    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c33c19b-0057-40dd-af41-fe69990fb374-tls-certs\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.094609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.080533    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-dshm\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.105564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.105070    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-tmp-dir\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.106097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.106069    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-model-cache\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.106480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.106443    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-home\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.107075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.106981    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-dshm\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.108947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.108922    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c33c19b-0057-40dd-af41-fe69990fb374-tls-certs\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.132802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.132774    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9wbw\" (UniqueName: \"kubernetes.io/projected/4c33c19b-0057-40dd-af41-fe69990fb374-kube-api-access-k9wbw\") pod \"test-model-0b1c15-kserve-7756859875-snm9k\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") " pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.269568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.268837    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:40.714192 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:27:40.714158    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c33c19b_0057_40dd_af41_fe69990fb374.slice/crio-251ae85fa071b9bbe24c004be3bf747cc6d357996a68d822cae369767adc5ea4 WatchSource:0}: Error finding container 251ae85fa071b9bbe24c004be3bf747cc6d357996a68d822cae369767adc5ea4: Status 404 returned error can't find the container with id 251ae85fa071b9bbe24c004be3bf747cc6d357996a68d822cae369767adc5ea4
Oct 30 14:27:40.714723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.714698    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"]
Oct 30 14:27:40.853691 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.853662    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:27:40.854551 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.854490    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:27:40.863474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:40.863433    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:27:41.037635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.035573    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k" event={"ID":"4c33c19b-0057-40dd-af41-fe69990fb374","Type":"ContainerStarted","Data":"faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f"}
Oct 30 14:27:41.037635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.035633    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k" event={"ID":"4c33c19b-0057-40dd-af41-fe69990fb374","Type":"ContainerStarted","Data":"251ae85fa071b9bbe24c004be3bf747cc6d357996a68d822cae369767adc5ea4"}
Oct 30 14:27:41.094448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.094378    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k" podStartSLOduration=2.094359016 podStartE2EDuration="2.094359016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:41.081804251 +0000 UTC m=+2326.113703191" watchObservedRunningTime="2026-10-30 14:27:41.094359016 +0000 UTC m=+2326.126257956"
Oct 30 14:27:41.256663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.256628    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"]
Oct 30 14:27:41.260299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.260272    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.270163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.270140    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:41.281772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.281746    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-acefbb-openshift-default-dockercfg-6v2bk\""
Oct 30 14:27:41.289700 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.288601    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"]
Oct 30 14:27:41.324105 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.324075    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istiod-ca-cert\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.324282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.324116    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-token\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.324282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.324137    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-credential-socket\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.324282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.324255    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-certs\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.324488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.324326    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-podinfo\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.324488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.324428    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-socket\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.324654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.324636    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8mxm\" (UniqueName: \"kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-kube-api-access-d8mxm\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.324720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.324679    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-envoy\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.324801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.324785    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-data\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.430566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.428238    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8mxm\" (UniqueName: \"kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-kube-api-access-d8mxm\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.430566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.428297    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-envoy\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.430566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.428362    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-data\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.430566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.428456    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istiod-ca-cert\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.430566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.428491    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-token\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.430566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.428521    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-credential-socket\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.430566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.428575    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-certs\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.430566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.428690    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-podinfo\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.430566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.428743    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-socket\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.430566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.429275    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-socket\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.462563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.441925    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-data\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.462563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.442622    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istiod-ca-cert\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.462563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.443207    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-credential-socket\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.462563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.443488    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-certs\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.480565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.463333    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-envoy\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.494572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.490699    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-podinfo\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.526565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.496690    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8mxm\" (UniqueName: \"kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-kube-api-access-d8mxm\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.526565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.522202    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-token\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.526565 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:27:41.524090    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:27:41.595855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.595818    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:41.953400 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:27:41.953366    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebfd942_253e_4b76_aeb4_6b5bcd0b40d4.slice/crio-5ed1104ddf16d2b8ce750336cbbd88ea88912449aa32d4911f5e532b6b0a8bc0 WatchSource:0}: Error finding container 5ed1104ddf16d2b8ce750336cbbd88ea88912449aa32d4911f5e532b6b0a8bc0: Status 404 returned error can't find the container with id 5ed1104ddf16d2b8ce750336cbbd88ea88912449aa32d4911f5e532b6b0a8bc0
Oct 30 14:27:41.956469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.956443    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"]
Oct 30 14:27:41.957188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.957154    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:27:41.957261 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.957225    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:27:41.957261 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:41.957253    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:27:42.045333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:42.045297    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" event={"ID":"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4","Type":"ContainerStarted","Data":"5ed1104ddf16d2b8ce750336cbbd88ea88912449aa32d4911f5e532b6b0a8bc0"}
Oct 30 14:27:43.054902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:43.054866    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" event={"ID":"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4","Type":"ContainerStarted","Data":"2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58"}
Oct 30 14:27:43.141967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:43.141915    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" podStartSLOduration=2.141898547 podStartE2EDuration="2.141898547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:43.124849185 +0000 UTC m=+2328.156748136" watchObservedRunningTime="2026-10-30 14:27:43.141898547 +0000 UTC m=+2328.173797486"
Oct 30 14:27:43.596818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:43.596785    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:43.602741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:43.602716    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:44.062494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:44.062455    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:44.063343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:44.063325    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:27:50.269506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:50.269467    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:50.282498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:50.282466    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:51.198989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:51.198961    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:27:53.207056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.207022    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"]
Oct 30 14:27:53.212391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.212361    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.225989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.225962    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"]
Oct 30 14:27:53.234554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.234516    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fde75d\"/\"kube-root-ca.crt\""
Oct 30 14:27:53.234697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.234584    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fde75d\"/\"default-dockercfg-n7lnd\""
Oct 30 14:27:53.243897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.243876    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fde75d\"/\"openshift-service-ca.crt\""
Oct 30 14:27:53.244037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.243902    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fde75d\"/\"test-model-fde75d-kserve-self-signed-certs\""
Oct 30 14:27:53.347828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.347792    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tmp-dir\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.347999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.347836    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-model-cache\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.348046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.348012    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tls-certs\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.348174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.348155    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-dshm\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.348212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.348188    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cgbt\" (UniqueName: \"kubernetes.io/projected/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-kube-api-access-9cgbt\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.348293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.348280    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-home\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.451952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.451906    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tls-certs\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.452391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.452367    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-dshm\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.452526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.452510    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cgbt\" (UniqueName: \"kubernetes.io/projected/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-kube-api-access-9cgbt\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.452733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.452716    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-home\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.452950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.452934    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tmp-dir\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.453079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.453064    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-model-cache\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.453831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.453808    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-model-cache\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.454859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.454833    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-home\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.455463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.455439    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tmp-dir\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.472241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.472173    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tls-certs\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.477858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.477796    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-dshm\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.508488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.508451    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cgbt\" (UniqueName: \"kubernetes.io/projected/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-kube-api-access-9cgbt\") pod \"test-model-fde75d-kserve-64b99544b5-rptrq\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") " pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.537598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.537299    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:27:53.880492 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:27:53.880456    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf18d355c_edfe_4d27_bb1b_ff09d18fd43e.slice/crio-61cbcab851e3110e6a72db020987b6c0a1fdc3935be35394697f6242547a04ce WatchSource:0}: Error finding container 61cbcab851e3110e6a72db020987b6c0a1fdc3935be35394697f6242547a04ce: Status 404 returned error can't find the container with id 61cbcab851e3110e6a72db020987b6c0a1fdc3935be35394697f6242547a04ce
Oct 30 14:27:53.885755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:53.885298    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"]
Oct 30 14:27:54.221685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:54.221600    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq" event={"ID":"f18d355c-edfe-4d27-bb1b-ff09d18fd43e","Type":"ContainerStarted","Data":"e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3"}
Oct 30 14:27:54.221685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:54.221653    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq" event={"ID":"f18d355c-edfe-4d27-bb1b-ff09d18fd43e","Type":"ContainerStarted","Data":"61cbcab851e3110e6a72db020987b6c0a1fdc3935be35394697f6242547a04ce"}
Oct 30 14:27:54.268174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:54.268123    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq" podStartSLOduration=1.2681102069999999 podStartE2EDuration="1.268110207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:54.267037584 +0000 UTC m=+2339.298936536" watchObservedRunningTime="2026-10-30 14:27:54.268110207 +0000 UTC m=+2339.300009172"
Oct 30 14:27:54.538265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:27:54.538188    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:28:03.538818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:03.538783    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:28:03.551730 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:03.551705    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:28:04.313251 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:04.313219    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:28:11.369613 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:28:11.369578    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:28:13.677595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.677556    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"]
Oct 30 14:28:13.683633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.683610    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.706698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.706656    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"]
Oct 30 14:28:13.854471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.854429    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1d3696-openshift-default-dockercfg-5g6bv\""
Oct 30 14:28:13.858750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.858724    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-token\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.863800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.860067    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-data\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.908566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.884164    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-credential-socket\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.908566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.884246    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-podinfo\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.908566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.884426    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v7jx\" (UniqueName: \"kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-kube-api-access-7v7jx\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.908566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.884588    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istiod-ca-cert\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.908566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.884629    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-envoy\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.908566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.884672    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-certs\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.908566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.884853    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-socket\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.987652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.987530    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-credential-socket\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.987911 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.987889    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-podinfo\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.988053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.988037    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v7jx\" (UniqueName: \"kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-kube-api-access-7v7jx\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.988208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.988188    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istiod-ca-cert\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.988323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.988308    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-envoy\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.988449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.988435    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-certs\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.988728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.988707    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-socket\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.989039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.989018    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-token\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.989202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.989184    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-data\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.989897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.989829    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istiod-ca-cert\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.990034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.990016    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-data\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.990406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.990386    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-certs\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.990528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.990404    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-socket\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:13.991816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:13.991787    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-credential-socket\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:14.009257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:14.009192    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-envoy\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:14.009759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:14.009732    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-podinfo\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:14.018247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:14.018177    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-token\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:14.020412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:14.020384    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v7jx\" (UniqueName: \"kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-kube-api-access-7v7jx\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:14.303215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:14.303186    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:14.689706 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:28:14.689669    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a91e56c_d22c_40a4_a8f1_e144eece82d4.slice/crio-3a1efd0b76abc881886e99cbe25d5e4136e9b88612b1350078ef674532dcf9af WatchSource:0}: Error finding container 3a1efd0b76abc881886e99cbe25d5e4136e9b88612b1350078ef674532dcf9af: Status 404 returned error can't find the container with id 3a1efd0b76abc881886e99cbe25d5e4136e9b88612b1350078ef674532dcf9af
Oct 30 14:28:14.691896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:14.691868    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"]
Oct 30 14:28:14.692907 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:14.692868    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:28:14.692982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:14.692943    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:28:14.692982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:14.692974    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:28:15.395555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:15.395508    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" event={"ID":"7a91e56c-d22c-40a4-a8f1-e144eece82d4","Type":"ContainerStarted","Data":"6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607"}
Oct 30 14:28:15.395739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:15.395590    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" event={"ID":"7a91e56c-d22c-40a4-a8f1-e144eece82d4","Type":"ContainerStarted","Data":"3a1efd0b76abc881886e99cbe25d5e4136e9b88612b1350078ef674532dcf9af"}
Oct 30 14:28:15.462817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:15.462768    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" podStartSLOduration=2.46275265 podStartE2EDuration="2.46275265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:28:15.462714598 +0000 UTC m=+2360.494613537" watchObservedRunningTime="2026-10-30 14:28:15.46275265 +0000 UTC m=+2360.494651589"
Oct 30 14:28:16.303492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:16.303460    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:16.310212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:16.310186    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:16.403661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:16.403619    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:16.404777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:16.404752    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:28:27.101155 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:27.101110    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"]
Oct 30 14:28:41.368794 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:28:41.368715    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:28:42.124467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.124424    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj" podUID="31152392-0c90-49cc-8365-14ffd1037fb7" containerName="main" containerID="cri-o://305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a" gracePeriod=45
Oct 30 14:28:42.401669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.401640    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:28:42.531922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.531883    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-model-cache\") pod \"31152392-0c90-49cc-8365-14ffd1037fb7\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") "
Oct 30 14:28:42.532158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.532121    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31152392-0c90-49cc-8365-14ffd1037fb7-kube-api-access-5twtd\" (UniqueName: \"kubernetes.io/projected/31152392-0c90-49cc-8365-14ffd1037fb7-kube-api-access-5twtd\") pod \"31152392-0c90-49cc-8365-14ffd1037fb7\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") "
Oct 30 14:28:42.532247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.532161    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-model-cache" pod "31152392-0c90-49cc-8365-14ffd1037fb7" (UID: "31152392-0c90-49cc-8365-14ffd1037fb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:28:42.532318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.532263    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31152392-0c90-49cc-8365-14ffd1037fb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/31152392-0c90-49cc-8365-14ffd1037fb7-tls-certs\") pod \"31152392-0c90-49cc-8365-14ffd1037fb7\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") "
Oct 30 14:28:42.532318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.532296    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-home\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-home\") pod \"31152392-0c90-49cc-8365-14ffd1037fb7\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") "
Oct 30 14:28:42.532451 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.532355    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-tmp-dir\") pod \"31152392-0c90-49cc-8365-14ffd1037fb7\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") "
Oct 30 14:28:42.532451 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.532384    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-dshm\") pod \"31152392-0c90-49cc-8365-14ffd1037fb7\" (UID: \"31152392-0c90-49cc-8365-14ffd1037fb7\") "
Oct 30 14:28:42.532615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.532520    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-home" pod "31152392-0c90-49cc-8365-14ffd1037fb7" (UID: "31152392-0c90-49cc-8365-14ffd1037fb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:28:42.532717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.532691    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-tmp-dir" pod "31152392-0c90-49cc-8365-14ffd1037fb7" (UID: "31152392-0c90-49cc-8365-14ffd1037fb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:28:42.533063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.533039    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:28:42.533063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.533061    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:28:42.533223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.533076    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:28:42.534472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.534448    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31152392-0c90-49cc-8365-14ffd1037fb7-tls-certs" pod "31152392-0c90-49cc-8365-14ffd1037fb7" (UID: "31152392-0c90-49cc-8365-14ffd1037fb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:28:42.534590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.534524    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31152392-0c90-49cc-8365-14ffd1037fb7-kube-api-access-5twtd" pod "31152392-0c90-49cc-8365-14ffd1037fb7" (UID: "31152392-0c90-49cc-8365-14ffd1037fb7"). InnerVolumeSpecName "kube-api-access-5twtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:28:42.534992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.534974    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-dshm" pod "31152392-0c90-49cc-8365-14ffd1037fb7" (UID: "31152392-0c90-49cc-8365-14ffd1037fb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:28:42.634231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.634171    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31152392-0c90-49cc-8365-14ffd1037fb7-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:28:42.634231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.634196    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5twtd\" (UniqueName: \"kubernetes.io/projected/31152392-0c90-49cc-8365-14ffd1037fb7-kube-api-access-5twtd\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:28:42.634231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.634206    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31152392-0c90-49cc-8365-14ffd1037fb7-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:28:42.648595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.648557    4530 generic.go:356] "Generic (PLEG): container finished" podID="31152392-0c90-49cc-8365-14ffd1037fb7" containerID="305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a" exitCode=0
Oct 30 14:28:42.648709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.648615    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"
Oct 30 14:28:42.648709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.648648    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj" event={"ID":"31152392-0c90-49cc-8365-14ffd1037fb7","Type":"ContainerDied","Data":"305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a"}
Oct 30 14:28:42.648709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.648704    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj" event={"ID":"31152392-0c90-49cc-8365-14ffd1037fb7","Type":"ContainerDied","Data":"85c5a2a46ab14977d58b25fda91a88daed69bdb6592f86ec833902ac463ca84c"}
Oct 30 14:28:42.648881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.648727    4530 scope.go:122] "RemoveContainer" containerID="305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a"
Oct 30 14:28:42.660496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.660481    4530 scope.go:122] "RemoveContainer" containerID="305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a"
Oct 30 14:28:42.660850 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:28:42.660832    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a\": container with ID starting with 305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a not found: ID does not exist" containerID="305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a"
Oct 30 14:28:42.660902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.660858    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a"} err="failed to get container status \"305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a\": rpc error: code = NotFound desc = could not find container \"305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a\": container with ID starting with 305651dd7b79ce4b463ed19aa53a289fa91e947c09e4f47f7ef65108c913944a not found: ID does not exist"
Oct 30 14:28:42.710884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.710856    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"]
Oct 30 14:28:42.716665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:42.716641    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-648d5f/test-model-648d5f-kserve-646847897c-zfbsj"]
Oct 30 14:28:43.117126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:43.117093    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31152392-0c90-49cc-8365-14ffd1037fb7" path="/var/lib/kubelet/pods/31152392-0c90-49cc-8365-14ffd1037fb7/volumes"
Oct 30 14:28:47.438339 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:47.438301    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:28:55.858677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:55.858647    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:28:55.859975 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:55.859948    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:28:55.863840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:55.863813    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:28:55.864698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:55.864675    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:28:55.864823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:55.864808    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:28:55.865823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:55.865807    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:28:56.459350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:28:56.459314    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"]
Oct 30 14:29:07.324020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:07.323988    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"]
Oct 30 14:29:11.368480 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:29:11.368441    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:29:11.481934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.481873    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5" podUID="b92c9a57-aa24-48a9-8278-cca33d6ad220" containerName="main" containerID="cri-o://793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e" gracePeriod=45
Oct 30 14:29:11.761109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.761080    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:29:11.791354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.791322    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-model-cache\") pod \"b92c9a57-aa24-48a9-8278-cca33d6ad220\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") "
Oct 30 14:29:11.791525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.791383    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-home\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-home\") pod \"b92c9a57-aa24-48a9-8278-cca33d6ad220\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") "
Oct 30 14:29:11.791525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.791418    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-tmp-dir\") pod \"b92c9a57-aa24-48a9-8278-cca33d6ad220\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") "
Oct 30 14:29:11.791675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.791656    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-model-cache" pod "b92c9a57-aa24-48a9-8278-cca33d6ad220" (UID: "b92c9a57-aa24-48a9-8278-cca33d6ad220"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:29:11.791739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.791713    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-dshm\") pod \"b92c9a57-aa24-48a9-8278-cca33d6ad220\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") "
Oct 30 14:29:11.791739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.791719    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-home" pod "b92c9a57-aa24-48a9-8278-cca33d6ad220" (UID: "b92c9a57-aa24-48a9-8278-cca33d6ad220"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:29:11.791814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.791744    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-tmp-dir" pod "b92c9a57-aa24-48a9-8278-cca33d6ad220" (UID: "b92c9a57-aa24-48a9-8278-cca33d6ad220"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:29:11.791857 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.791809    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b92c9a57-aa24-48a9-8278-cca33d6ad220-tls-certs\" (UniqueName: \"kubernetes.io/secret/b92c9a57-aa24-48a9-8278-cca33d6ad220-tls-certs\") pod \"b92c9a57-aa24-48a9-8278-cca33d6ad220\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") "
Oct 30 14:29:11.791857 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.791845    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92c9a57-aa24-48a9-8278-cca33d6ad220-kube-api-access-5bgz2\" (UniqueName: \"kubernetes.io/projected/b92c9a57-aa24-48a9-8278-cca33d6ad220-kube-api-access-5bgz2\") pod \"b92c9a57-aa24-48a9-8278-cca33d6ad220\" (UID: \"b92c9a57-aa24-48a9-8278-cca33d6ad220\") "
Oct 30 14:29:11.792472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.792450    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:29:11.792472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.792470    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:29:11.792472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.792478    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:29:11.794128 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.794099    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b92c9a57-aa24-48a9-8278-cca33d6ad220-tls-certs" pod "b92c9a57-aa24-48a9-8278-cca33d6ad220" (UID: "b92c9a57-aa24-48a9-8278-cca33d6ad220"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:29:11.794299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.794282    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-dshm" pod "b92c9a57-aa24-48a9-8278-cca33d6ad220" (UID: "b92c9a57-aa24-48a9-8278-cca33d6ad220"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:29:11.794432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.794413    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92c9a57-aa24-48a9-8278-cca33d6ad220-kube-api-access-5bgz2" pod "b92c9a57-aa24-48a9-8278-cca33d6ad220" (UID: "b92c9a57-aa24-48a9-8278-cca33d6ad220"). InnerVolumeSpecName "kube-api-access-5bgz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:29:11.889308 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.889271    4530 generic.go:356] "Generic (PLEG): container finished" podID="b92c9a57-aa24-48a9-8278-cca33d6ad220" containerID="793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e" exitCode=0
Oct 30 14:29:11.889308 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.889308    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"
Oct 30 14:29:11.889514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.889355    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5" event={"ID":"b92c9a57-aa24-48a9-8278-cca33d6ad220","Type":"ContainerDied","Data":"793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e"}
Oct 30 14:29:11.889514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.889399    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5" event={"ID":"b92c9a57-aa24-48a9-8278-cca33d6ad220","Type":"ContainerDied","Data":"e2b41b2752b93f038547352efb0f8112d8fcbc6b93280a9d8500f828e11a4be8"}
Oct 30 14:29:11.889514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.889420    4530 scope.go:122] "RemoveContainer" containerID="793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e"
Oct 30 14:29:11.893569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.893530    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b92c9a57-aa24-48a9-8278-cca33d6ad220-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:29:11.893693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.893573    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92c9a57-aa24-48a9-8278-cca33d6ad220-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:29:11.893693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.893586    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bgz2\" (UniqueName: \"kubernetes.io/projected/b92c9a57-aa24-48a9-8278-cca33d6ad220-kube-api-access-5bgz2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:29:11.904406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.904382    4530 scope.go:122] "RemoveContainer" containerID="793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e"
Oct 30 14:29:11.904688 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:29:11.904666    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e\": container with ID starting with 793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e not found: ID does not exist" containerID="793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e"
Oct 30 14:29:11.904760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.904697    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e"} err="failed to get container status \"793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e\": rpc error: code = NotFound desc = could not find container \"793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e\": container with ID starting with 793afc5e6b9800f89ff632cdd0fc8784f32bced84132f2e2c5fd81a26ea7eb6e not found: ID does not exist"
Oct 30 14:29:11.952905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.952828    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"]
Oct 30 14:29:11.958301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:11.958278    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a9542c/test-model-a9542c-kserve-6595554bd7-bbvt5"]
Oct 30 14:29:12.324968 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:12.324926    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc" podUID="376dbba0-6d14-4351-811a-9d9b82c7b8cc" containerName="payload-pre-processing" containerID="cri-o://36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286" gracePeriod=25
Oct 30 14:29:13.152080 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:13.152040    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92c9a57-aa24-48a9-8278-cca33d6ad220" path="/var/lib/kubelet/pods/b92c9a57-aa24-48a9-8278-cca33d6ad220/volumes"
Oct 30 14:29:19.121957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:19.121921    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc" podUID="376dbba0-6d14-4351-811a-9d9b82c7b8cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:29:22.505699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:22.505656    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" podUID="e329b8d309e1f8141c44f91d4bb26de8" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:29:29.121803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:29.121763    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc" podUID="376dbba0-6d14-4351-811a-9d9b82c7b8cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:29:29.699994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:29.699042    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": read tcp 10.133.0.2:48946->10.133.0.238:15021: read: connection reset by peer" start-of-body=
Oct 30 14:29:29.699994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:29.699127    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": read tcp 10.133.0.2:48946->10.133.0.238:15021: read: connection reset by peer"
Oct 30 14:29:30.086987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:30.086958    4530 generic.go:356] "Generic (PLEG): container finished" podID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerID="2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58" exitCode=0
Oct 30 14:29:30.087163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:30.087027    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" event={"ID":"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4","Type":"ContainerDied","Data":"2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58"}
Oct 30 14:29:30.087484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:30.087467    4530 scope.go:122] "RemoveContainer" containerID="2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58"
Oct 30 14:29:30.089340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:30.089310    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:29:30.089470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:30.089385    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:29:30.089470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:30.089427    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:29:31.098241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:31.098208    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" event={"ID":"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4","Type":"ContainerStarted","Data":"36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685"}
Oct 30 14:29:31.598564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:31.597919    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:29:31.598564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:31.598098    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 14:29:31.598564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:31.598144    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 14:29:32.603616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:32.602895    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 14:29:32.603616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:32.602960    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 14:29:32.657428 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:32.657395    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"
Oct 30 14:29:32.738889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:32.738856    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/376dbba0-6d14-4351-811a-9d9b82c7b8cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376dbba0-6d14-4351-811a-9d9b82c7b8cc-plugins-config-volume\") pod \"376dbba0-6d14-4351-811a-9d9b82c7b8cc\" (UID: \"376dbba0-6d14-4351-811a-9d9b82c7b8cc\") "
Oct 30 14:29:32.739080 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:32.739062    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376dbba0-6d14-4351-811a-9d9b82c7b8cc-kube-api-access-gp2g8\" (UniqueName: \"kubernetes.io/projected/376dbba0-6d14-4351-811a-9d9b82c7b8cc-kube-api-access-gp2g8\") pod \"376dbba0-6d14-4351-811a-9d9b82c7b8cc\" (UID: \"376dbba0-6d14-4351-811a-9d9b82c7b8cc\") "
Oct 30 14:29:32.739366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:32.739337    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376dbba0-6d14-4351-811a-9d9b82c7b8cc-plugins-config-volume" pod "376dbba0-6d14-4351-811a-9d9b82c7b8cc" (UID: "376dbba0-6d14-4351-811a-9d9b82c7b8cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:29:32.739697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:32.739677    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376dbba0-6d14-4351-811a-9d9b82c7b8cc-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:29:32.741647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:32.741615    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376dbba0-6d14-4351-811a-9d9b82c7b8cc-kube-api-access-gp2g8" pod "376dbba0-6d14-4351-811a-9d9b82c7b8cc" (UID: "376dbba0-6d14-4351-811a-9d9b82c7b8cc"). InnerVolumeSpecName "kube-api-access-gp2g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:29:32.843690 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:32.843656    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp2g8\" (UniqueName: \"kubernetes.io/projected/376dbba0-6d14-4351-811a-9d9b82c7b8cc-kube-api-access-gp2g8\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:29:33.123311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:33.123274    4530 generic.go:356] "Generic (PLEG): container finished" podID="376dbba0-6d14-4351-811a-9d9b82c7b8cc" containerID="36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286" exitCode=0
Oct 30 14:29:33.123526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:33.123333    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"
Oct 30 14:29:33.123619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:33.123597    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc" event={"ID":"376dbba0-6d14-4351-811a-9d9b82c7b8cc","Type":"ContainerDied","Data":"36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286"}
Oct 30 14:29:33.123678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:33.123649    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc" event={"ID":"376dbba0-6d14-4351-811a-9d9b82c7b8cc","Type":"ContainerDied","Data":"7367ce86e6a11226f061709c2cecf9682945a3a99e89d29a04916628f2718cdd"}
Oct 30 14:29:33.123732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:33.123680    4530 scope.go:122] "RemoveContainer" containerID="36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286"
Oct 30 14:29:33.142263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:33.142240    4530 scope.go:122] "RemoveContainer" containerID="36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286"
Oct 30 14:29:33.142521 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:29:33.142500    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286\": container with ID starting with 36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286 not found: ID does not exist" containerID="36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286"
Oct 30 14:29:33.142648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:33.142528    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286"} err="failed to get container status \"36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286\": rpc error: code = NotFound desc = could not find container \"36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286\": container with ID starting with 36d5b33580a11a568014de7b36d7895d278abf697f3b4821ed08d9426e6e3286 not found: ID does not exist"
Oct 30 14:29:33.209795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:33.209765    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"]
Oct 30 14:29:33.219165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:33.219136    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-648d5f-7fc79c5b59-j4bjc"]
Oct 30 14:29:33.606071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:33.606039    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 14:29:33.606581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:33.606432    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 14:29:34.604567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:34.603650    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 14:29:34.604567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:34.603720    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 14:29:35.119968 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:35.119935    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="376dbba0-6d14-4351-811a-9d9b82c7b8cc" path="/var/lib/kubelet/pods/376dbba0-6d14-4351-811a-9d9b82c7b8cc/volumes"
Oct 30 14:29:35.601676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:35.601637    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 14:29:35.601983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:35.601949    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 14:29:36.604567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:36.598883    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 14:29:36.604567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:36.598960    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 14:29:37.601477 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:37.601445    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:29:37.601993 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:37.601970    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:29:37.602677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:37.602658    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:29:41.368397 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:29:41.368361    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:29:47.317913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:47.317879    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"]
Oct 30 14:29:47.318417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:47.318065    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf" podUID="9446e240-0a9f-4fcd-8f73-0a8ece3aa52f" containerName="maas-api" containerID="cri-o://568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1" gracePeriod=30
Oct 30 14:29:47.723956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:47.723929    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:29:47.830613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:47.805227    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-maas-api-tls\") pod \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") "
Oct 30 14:29:47.830613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:47.805358    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-metrics-certs\") pod \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") "
Oct 30 14:29:47.830613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:47.829113    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-openshift-service-ca" pod "9446e240-0a9f-4fcd-8f73-0a8ece3aa52f" (UID: "9446e240-0a9f-4fcd-8f73-0a8ece3aa52f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:29:47.830613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:47.805498    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-openshift-service-ca\") pod \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") "
Oct 30 14:29:47.830613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:47.829292    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-kube-api-access-v8jrk\" (UniqueName: \"kubernetes.io/projected/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-kube-api-access-v8jrk\") pod \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\" (UID: \"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f\") "
Oct 30 14:29:47.830613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:47.830173    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:29:47.953648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:47.953524    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-maas-api-tls" pod "9446e240-0a9f-4fcd-8f73-0a8ece3aa52f" (UID: "9446e240-0a9f-4fcd-8f73-0a8ece3aa52f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:29:47.966309 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:47.965095    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-metrics-certs" pod "9446e240-0a9f-4fcd-8f73-0a8ece3aa52f" (UID: "9446e240-0a9f-4fcd-8f73-0a8ece3aa52f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:29:47.980230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:47.979978    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-kube-api-access-v8jrk" pod "9446e240-0a9f-4fcd-8f73-0a8ece3aa52f" (UID: "9446e240-0a9f-4fcd-8f73-0a8ece3aa52f"). InnerVolumeSpecName "kube-api-access-v8jrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:29:48.045660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:48.045627    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:29:48.045660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:48.045657    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8jrk\" (UniqueName: \"kubernetes.io/projected/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-kube-api-access-v8jrk\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:29:48.045872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:48.045669    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:29:48.253881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:48.253792    4530 generic.go:356] "Generic (PLEG): container finished" podID="9446e240-0a9f-4fcd-8f73-0a8ece3aa52f" containerID="568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1" exitCode=0
Oct 30 14:29:48.253881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:48.253843    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"
Oct 30 14:29:48.254097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:48.253876    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf" event={"ID":"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f","Type":"ContainerDied","Data":"568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1"}
Oct 30 14:29:48.254097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:48.253928    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf" event={"ID":"9446e240-0a9f-4fcd-8f73-0a8ece3aa52f","Type":"ContainerDied","Data":"2e694074b21cafa846883f47f490295e74b9a268cc93d45f9f192d465967d375"}
Oct 30 14:29:48.254097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:48.253953    4530 scope.go:122] "RemoveContainer" containerID="568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1"
Oct 30 14:29:48.265395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:48.265377    4530 scope.go:122] "RemoveContainer" containerID="568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1"
Oct 30 14:29:48.265651 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:29:48.265629    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1\": container with ID starting with 568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1 not found: ID does not exist" containerID="568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1"
Oct 30 14:29:48.265703 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:48.265659    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1"} err="failed to get container status \"568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1\": rpc error: code = NotFound desc = could not find container \"568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1\": container with ID starting with 568d16bf6a3669e116dedefd967e0cbdade486208f015b8b1f509064d317aba1 not found: ID does not exist"
Oct 30 14:29:48.318092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:48.318065    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"]
Oct 30 14:29:48.332197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:48.332171    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-648d5f-6ff9547dd-m4nnf"]
Oct 30 14:29:49.116964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:49.116929    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9446e240-0a9f-4fcd-8f73-0a8ece3aa52f" path="/var/lib/kubelet/pods/9446e240-0a9f-4fcd-8f73-0a8ece3aa52f/volumes"
Oct 30 14:29:55.052979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:55.052933    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"]
Oct 30 14:29:56.647599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:56.647563    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"]
Oct 30 14:29:56.648042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:29:56.647814    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy" containerID="cri-o://3a79a63f9862f4638c8b440343112685b08245651e01afbff6ec7b16ef81a355" gracePeriod=30
Oct 30 14:30:00.254022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.253982    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"]
Oct 30 14:30:00.256171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.255099    4530 cpu_manager.go:414] "containerMap: removing container" podUID="376dbba0-6d14-4351-811a-9d9b82c7b8cc" containerName="payload-pre-processing"
Oct 30 14:30:00.256171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.255122    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376dbba0-6d14-4351-811a-9d9b82c7b8cc" containerName="payload-pre-processing"
Oct 30 14:30:00.256171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.255159    4530 cpu_manager.go:414] "containerMap: removing container" podUID="31152392-0c90-49cc-8365-14ffd1037fb7" containerName="main"
Oct 30 14:30:00.256171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.255167    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31152392-0c90-49cc-8365-14ffd1037fb7" containerName="main"
Oct 30 14:30:00.256171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.255188    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b92c9a57-aa24-48a9-8278-cca33d6ad220" containerName="main"
Oct 30 14:30:00.256171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.255197    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92c9a57-aa24-48a9-8278-cca33d6ad220" containerName="main"
Oct 30 14:30:00.256171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.255229    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9446e240-0a9f-4fcd-8f73-0a8ece3aa52f" containerName="maas-api"
Oct 30 14:30:00.256171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.255238    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9446e240-0a9f-4fcd-8f73-0a8ece3aa52f" containerName="maas-api"
Oct 30 14:30:00.256171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.255413    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="376dbba0-6d14-4351-811a-9d9b82c7b8cc" containerName="payload-pre-processing"
Oct 30 14:30:00.256171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.255434    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="31152392-0c90-49cc-8365-14ffd1037fb7" containerName="main"
Oct 30 14:30:00.256171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.255448    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92c9a57-aa24-48a9-8278-cca33d6ad220" containerName="main"
Oct 30 14:30:00.256171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.255462    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9446e240-0a9f-4fcd-8f73-0a8ece3aa52f" containerName="maas-api"
Oct 30 14:30:00.261583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.261556    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"
Oct 30 14:30:00.285424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.285397    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"]
Oct 30 14:30:00.291213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.291187    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"
Oct 30 14:30:00.291833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.291807    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"]
Oct 30 14:30:00.300407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.300378    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:30:00.300533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.300425    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"]
Oct 30 14:30:00.305960 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.305928    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"
Oct 30 14:30:00.310349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.310327    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"]
Oct 30 14:30:00.314268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.314242    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"]
Oct 30 14:30:00.347008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.346972    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss\" (UID: \"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"
Oct 30 14:30:00.347008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.347020    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qnt\" (UniqueName: \"kubernetes.io/projected/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-kube-api-access-68qnt\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb\" (UID: \"b90d5d9a-2c23-4cdf-9317-13f9803ca44b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"
Oct 30 14:30:00.347184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.347043    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg\" (UID: \"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"
Oct 30 14:30:00.347233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.347224    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb\" (UID: \"b90d5d9a-2c23-4cdf-9317-13f9803ca44b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"
Oct 30 14:30:00.347389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.347366    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmbrk\" (UniqueName: \"kubernetes.io/projected/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-kube-api-access-xmbrk\") pod \"maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg\" (UID: \"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"
Oct 30 14:30:00.347535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.347511    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q2hr\" (UniqueName: \"kubernetes.io/projected/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-kube-api-access-6q2hr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss\" (UID: \"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"
Oct 30 14:30:00.448827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.448793    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb\" (UID: \"b90d5d9a-2c23-4cdf-9317-13f9803ca44b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"
Oct 30 14:30:00.449002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.448908    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmbrk\" (UniqueName: \"kubernetes.io/projected/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-kube-api-access-xmbrk\") pod \"maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg\" (UID: \"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"
Oct 30 14:30:00.449002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.448985    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q2hr\" (UniqueName: \"kubernetes.io/projected/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-kube-api-access-6q2hr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss\" (UID: \"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"
Oct 30 14:30:00.449136 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.449055    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss\" (UID: \"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"
Oct 30 14:30:00.449198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.449114    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qnt\" (UniqueName: \"kubernetes.io/projected/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-kube-api-access-68qnt\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb\" (UID: \"b90d5d9a-2c23-4cdf-9317-13f9803ca44b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"
Oct 30 14:30:00.449434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.449183    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg\" (UID: \"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"
Oct 30 14:30:00.449695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.449675    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb\" (UID: \"b90d5d9a-2c23-4cdf-9317-13f9803ca44b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"
Oct 30 14:30:00.449798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.449777    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss\" (UID: \"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"
Oct 30 14:30:00.449940 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.449922    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg\" (UID: \"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"
Oct 30 14:30:00.476305 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.476273    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmbrk\" (UniqueName: \"kubernetes.io/projected/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-kube-api-access-xmbrk\") pod \"maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg\" (UID: \"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"
Oct 30 14:30:00.476408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.476308    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q2hr\" (UniqueName: \"kubernetes.io/projected/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-kube-api-access-6q2hr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss\" (UID: \"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"
Oct 30 14:30:00.476459 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.476424    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qnt\" (UniqueName: \"kubernetes.io/projected/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-kube-api-access-68qnt\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb\" (UID: \"b90d5d9a-2c23-4cdf-9317-13f9803ca44b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"
Oct 30 14:30:00.596022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.595994    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"
Oct 30 14:30:00.624438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.624414    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"
Oct 30 14:30:00.632717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.632682    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"
Oct 30 14:30:00.915119 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:30:00.915082    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd645d1ed_30a7_4229_bc89_7bdf9fbeb19b.slice/crio-f5128bbc8f2e8a3cc1ed6234bd5bbaa124f799ffab08b40346b0eb75fa2e90a3 WatchSource:0}: Error finding container f5128bbc8f2e8a3cc1ed6234bd5bbaa124f799ffab08b40346b0eb75fa2e90a3: Status 404 returned error can't find the container with id f5128bbc8f2e8a3cc1ed6234bd5bbaa124f799ffab08b40346b0eb75fa2e90a3
Oct 30 14:30:00.916917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.916894    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"]
Oct 30 14:30:00.955560 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:30:00.955521    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90d5d9a_2c23_4cdf_9317_13f9803ca44b.slice/crio-af36c8a65085b513bdefe135e464a1d88a70f2d2fb90545039e9561d6ed91f05 WatchSource:0}: Error finding container af36c8a65085b513bdefe135e464a1d88a70f2d2fb90545039e9561d6ed91f05: Status 404 returned error can't find the container with id af36c8a65085b513bdefe135e464a1d88a70f2d2fb90545039e9561d6ed91f05
Oct 30 14:30:00.959701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.959677    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"]
Oct 30 14:30:00.968104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:00.968078    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"]
Oct 30 14:30:00.968827 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:30:00.968805    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a6c9f1_8d5c_4c57_921f_1fe2b65cdc9f.slice/crio-5fe2a0b6abf01a478e3503978cb644fdaa343a9808fe75781e758344705b2ab7 WatchSource:0}: Error finding container 5fe2a0b6abf01a478e3503978cb644fdaa343a9808fe75781e758344705b2ab7: Status 404 returned error can't find the container with id 5fe2a0b6abf01a478e3503978cb644fdaa343a9808fe75781e758344705b2ab7
Oct 30 14:30:01.446786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:01.442649    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb" event={"ID":"b90d5d9a-2c23-4cdf-9317-13f9803ca44b","Type":"ContainerStarted","Data":"af36c8a65085b513bdefe135e464a1d88a70f2d2fb90545039e9561d6ed91f05"}
Oct 30 14:30:01.446786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:01.444808    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss" event={"ID":"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b","Type":"ContainerStarted","Data":"7902c23be7ce3e42bcd899a8bbb2cf442936c95c6077eb710e6a010f343ff2e3"}
Oct 30 14:30:01.446786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:01.444850    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss" event={"ID":"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b","Type":"ContainerStarted","Data":"f5128bbc8f2e8a3cc1ed6234bd5bbaa124f799ffab08b40346b0eb75fa2e90a3"}
Oct 30 14:30:01.463636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:01.460415    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg" event={"ID":"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f","Type":"ContainerStarted","Data":"5fe2a0b6abf01a478e3503978cb644fdaa343a9808fe75781e758344705b2ab7"}
Oct 30 14:30:01.549594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:01.545932    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss" podStartSLOduration=1.545913484 podStartE2EDuration="1.545913484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:01.543715024 +0000 UTC m=+2466.575613964" watchObservedRunningTime="2026-10-30 14:30:01.545913484 +0000 UTC m=+2466.577812426"
Oct 30 14:30:02.012148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.012120    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:30:02.084062 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.084035    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b91f88-14b5-4321-bd83-5e752955ae3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b91f88-14b5-4321-bd83-5e752955ae3b-istiod-ca-cert\") pod \"07b91f88-14b5-4321-bd83-5e752955ae3b\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") "
Oct 30 14:30:02.084229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.084074    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-kube-api-access-mgf7h\" (UniqueName: \"kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-kube-api-access-mgf7h\") pod \"07b91f88-14b5-4321-bd83-5e752955ae3b\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") "
Oct 30 14:30:02.084229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.084118    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-credential-socket\") pod \"07b91f88-14b5-4321-bd83-5e752955ae3b\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") "
Oct 30 14:30:02.084229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.084160    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-podinfo\") pod \"07b91f88-14b5-4321-bd83-5e752955ae3b\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") "
Oct 30 14:30:02.084421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.084269    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-token\" (UniqueName: \"kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-token\") pod \"07b91f88-14b5-4321-bd83-5e752955ae3b\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") "
Oct 30 14:30:02.084421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.084410    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-envoy\") pod \"07b91f88-14b5-4321-bd83-5e752955ae3b\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") "
Oct 30 14:30:02.084551 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.084499    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-credential-socket" pod "07b91f88-14b5-4321-bd83-5e752955ae3b" (UID: "07b91f88-14b5-4321-bd83-5e752955ae3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:02.084627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.084564    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-socket\") pod \"07b91f88-14b5-4321-bd83-5e752955ae3b\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") "
Oct 30 14:30:02.084627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.084613    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-data\") pod \"07b91f88-14b5-4321-bd83-5e752955ae3b\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") "
Oct 30 14:30:02.084758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.084710    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-certs\") pod \"07b91f88-14b5-4321-bd83-5e752955ae3b\" (UID: \"07b91f88-14b5-4321-bd83-5e752955ae3b\") "
Oct 30 14:30:02.085045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.085023    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-socket" pod "07b91f88-14b5-4321-bd83-5e752955ae3b" (UID: "07b91f88-14b5-4321-bd83-5e752955ae3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:02.085195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.085092    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-data" pod "07b91f88-14b5-4321-bd83-5e752955ae3b" (UID: "07b91f88-14b5-4321-bd83-5e752955ae3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:02.085272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.085241    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-certs" pod "07b91f88-14b5-4321-bd83-5e752955ae3b" (UID: "07b91f88-14b5-4321-bd83-5e752955ae3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:02.085552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.085516    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:02.085653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.085562    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:02.085653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.085579    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:02.085653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.085593    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:02.085844 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.085759    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b91f88-14b5-4321-bd83-5e752955ae3b-istiod-ca-cert" pod "07b91f88-14b5-4321-bd83-5e752955ae3b" (UID: "07b91f88-14b5-4321-bd83-5e752955ae3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:30:02.086666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.086633    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-podinfo" pod "07b91f88-14b5-4321-bd83-5e752955ae3b" (UID: "07b91f88-14b5-4321-bd83-5e752955ae3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:30:02.086759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.086668    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-token" pod "07b91f88-14b5-4321-bd83-5e752955ae3b" (UID: "07b91f88-14b5-4321-bd83-5e752955ae3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:30:02.086862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.086846    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-envoy" pod "07b91f88-14b5-4321-bd83-5e752955ae3b" (UID: "07b91f88-14b5-4321-bd83-5e752955ae3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:02.087170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.087153    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-kube-api-access-mgf7h" pod "07b91f88-14b5-4321-bd83-5e752955ae3b" (UID: "07b91f88-14b5-4321-bd83-5e752955ae3b"). InnerVolumeSpecName "kube-api-access-mgf7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:30:02.185286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.185259    4530 scope.go:122] "RemoveContainer" containerID="453f189ce1475a9f8e93545e2c7cb3da5fbaf826b768c3649384461a46574c90"
Oct 30 14:30:02.186763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.186739    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b91f88-14b5-4321-bd83-5e752955ae3b-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:02.186763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.186765    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgf7h\" (UniqueName: \"kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-kube-api-access-mgf7h\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:02.186944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.186775    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:02.186944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.186783    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:02.186944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.186792    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b91f88-14b5-4321-bd83-5e752955ae3b-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:02.197349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.197329    4530 scope.go:122] "RemoveContainer" containerID="3a79a63f9862f4638c8b440343112685b08245651e01afbff6ec7b16ef81a355"
Oct 30 14:30:02.486521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.486439    4530 generic.go:356] "Generic (PLEG): container finished" podID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" containerID="5a7d12d291a22e6fee5ea89fe9f4358c6a03fcbdbfff6f890030e4872f65f641" exitCode=60
Oct 30 14:30:02.487007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.486555    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb" event={"ID":"b90d5d9a-2c23-4cdf-9317-13f9803ca44b","Type":"ContainerDied","Data":"5a7d12d291a22e6fee5ea89fe9f4358c6a03fcbdbfff6f890030e4872f65f641"}
Oct 30 14:30:02.487007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.486828    4530 scope.go:122] "RemoveContainer" containerID="5a7d12d291a22e6fee5ea89fe9f4358c6a03fcbdbfff6f890030e4872f65f641"
Oct 30 14:30:02.488896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.488817    4530 generic.go:356] "Generic (PLEG): container finished" podID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" containerID="7902c23be7ce3e42bcd899a8bbb2cf442936c95c6077eb710e6a010f343ff2e3" exitCode=60
Oct 30 14:30:02.488950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.488918    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss" event={"ID":"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b","Type":"ContainerDied","Data":"7902c23be7ce3e42bcd899a8bbb2cf442936c95c6077eb710e6a010f343ff2e3"}
Oct 30 14:30:02.489220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.489201    4530 scope.go:122] "RemoveContainer" containerID="7902c23be7ce3e42bcd899a8bbb2cf442936c95c6077eb710e6a010f343ff2e3"
Oct 30 14:30:02.494554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.494521    4530 generic.go:356] "Generic (PLEG): container finished" podID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" containerID="d71eeb61a1bdb354873898bd29565f62381efee16247ba11a2d4d78946b4240c" exitCode=60
Oct 30 14:30:02.494669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.494626    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"
Oct 30 14:30:02.494807 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.494782    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg" event={"ID":"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f","Type":"ContainerDied","Data":"d71eeb61a1bdb354873898bd29565f62381efee16247ba11a2d4d78946b4240c"}
Oct 30 14:30:02.495065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.494970    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" event={"ID":"07b91f88-14b5-4321-bd83-5e752955ae3b","Type":"ContainerDied","Data":"3a79a63f9862f4638c8b440343112685b08245651e01afbff6ec7b16ef81a355"}
Oct 30 14:30:02.495065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.495006    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh" event={"ID":"07b91f88-14b5-4321-bd83-5e752955ae3b","Type":"ContainerDied","Data":"765f3027701e9ae2edc354050a1a71432afba4a18d2fc48e321490f282300e7e"}
Oct 30 14:30:02.495065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.495010    4530 scope.go:122] "RemoveContainer" containerID="d71eeb61a1bdb354873898bd29565f62381efee16247ba11a2d4d78946b4240c"
Oct 30 14:30:02.660987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.660957    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"]
Oct 30 14:30:02.666049 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:02.666022    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-648d5f-openshift-default-86d675f7b9-dr4nh"]
Oct 30 14:30:03.117122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:03.117092    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" path="/var/lib/kubelet/pods/07b91f88-14b5-4321-bd83-5e752955ae3b/volumes"
Oct 30 14:30:03.506253 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:03.506222    4530 generic.go:356] "Generic (PLEG): container finished" podID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" containerID="43b7d2219c6914092450420c2e6f9075b36172b0213ac562dca8bfc590866b2e" exitCode=60
Oct 30 14:30:03.506719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:03.506303    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss" event={"ID":"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b","Type":"ContainerDied","Data":"43b7d2219c6914092450420c2e6f9075b36172b0213ac562dca8bfc590866b2e"}
Oct 30 14:30:03.506719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:03.506365    4530 scope.go:122] "RemoveContainer" containerID="7902c23be7ce3e42bcd899a8bbb2cf442936c95c6077eb710e6a010f343ff2e3"
Oct 30 14:30:03.506719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:03.506698    4530 scope.go:122] "RemoveContainer" containerID="43b7d2219c6914092450420c2e6f9075b36172b0213ac562dca8bfc590866b2e"
Oct 30 14:30:03.507073 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:03.507047    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss_odh-ai-gateway-infra(d645d1ed-30a7-4229-bc89-7bdf9fbeb19b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b"
Oct 30 14:30:03.508488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:03.508456    4530 generic.go:356] "Generic (PLEG): container finished" podID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" containerID="1ee1526f83190fb84706c23f5c58647e0a37fef2f384131536fb743e8094e482" exitCode=60
Oct 30 14:30:03.508610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:03.508534    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg" event={"ID":"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f","Type":"ContainerDied","Data":"1ee1526f83190fb84706c23f5c58647e0a37fef2f384131536fb743e8094e482"}
Oct 30 14:30:03.508809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:03.508793    4530 scope.go:122] "RemoveContainer" containerID="1ee1526f83190fb84706c23f5c58647e0a37fef2f384131536fb743e8094e482"
Oct 30 14:30:03.509070 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:03.509046    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg_odh-ai-gateway-infra(53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f"
Oct 30 14:30:03.510764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:03.510742    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb" event={"ID":"b90d5d9a-2c23-4cdf-9317-13f9803ca44b","Type":"ContainerStarted","Data":"7bf4559295f0a83d56eb1b4f3e4a87cf4d1aadc941ffac05f477d5a66ed4740d"}
Oct 30 14:30:03.607221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:03.607170    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb" podStartSLOduration=3.607157794 podStartE2EDuration="3.607157794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:03.603804402 +0000 UTC m=+2468.635703340" watchObservedRunningTime="2026-10-30 14:30:03.607157794 +0000 UTC m=+2468.639056733"
Oct 30 14:30:03.693027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:03.693004    4530 scope.go:122] "RemoveContainer" containerID="d71eeb61a1bdb354873898bd29565f62381efee16247ba11a2d4d78946b4240c"
Oct 30 14:30:04.521116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:04.521083    4530 generic.go:356] "Generic (PLEG): container finished" podID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" containerID="7bf4559295f0a83d56eb1b4f3e4a87cf4d1aadc941ffac05f477d5a66ed4740d" exitCode=60
Oct 30 14:30:04.521585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:04.521109    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb" event={"ID":"b90d5d9a-2c23-4cdf-9317-13f9803ca44b","Type":"ContainerDied","Data":"7bf4559295f0a83d56eb1b4f3e4a87cf4d1aadc941ffac05f477d5a66ed4740d"}
Oct 30 14:30:04.521585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:04.521174    4530 scope.go:122] "RemoveContainer" containerID="5a7d12d291a22e6fee5ea89fe9f4358c6a03fcbdbfff6f890030e4872f65f641"
Oct 30 14:30:04.521585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:04.521409    4530 scope.go:122] "RemoveContainer" containerID="7bf4559295f0a83d56eb1b4f3e4a87cf4d1aadc941ffac05f477d5a66ed4740d"
Oct 30 14:30:04.521726 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:04.521711    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb_odh-ai-gateway-infra(b90d5d9a-2c23-4cdf-9317-13f9803ca44b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b"
Oct 30 14:30:04.523435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:04.523418    4530 scope.go:122] "RemoveContainer" containerID="43b7d2219c6914092450420c2e6f9075b36172b0213ac562dca8bfc590866b2e"
Oct 30 14:30:04.523664 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:04.523645    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss_odh-ai-gateway-infra(d645d1ed-30a7-4229-bc89-7bdf9fbeb19b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b"
Oct 30 14:30:04.525079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:04.525064    4530 scope.go:122] "RemoveContainer" containerID="1ee1526f83190fb84706c23f5c58647e0a37fef2f384131536fb743e8094e482"
Oct 30 14:30:04.525312 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:04.525291    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg_odh-ai-gateway-infra(53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f"
Oct 30 14:30:05.536694 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:05.536644    4530 scope.go:122] "RemoveContainer" containerID="7bf4559295f0a83d56eb1b4f3e4a87cf4d1aadc941ffac05f477d5a66ed4740d"
Oct 30 14:30:05.537076 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:05.536958    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb_odh-ai-gateway-infra(b90d5d9a-2c23-4cdf-9317-13f9803ca44b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b"
Oct 30 14:30:07.160773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.160736    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"]
Oct 30 14:30:07.161925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.161887    4530 cpu_manager.go:414] "containerMap: removing container" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy"
Oct 30 14:30:07.161925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.161911    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy"
Oct 30 14:30:07.162114 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.161948    4530 cpu_manager.go:414] "containerMap: removing container" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy"
Oct 30 14:30:07.162114 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.161957    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy"
Oct 30 14:30:07.162184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.162120    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy"
Oct 30 14:30:07.167433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.167415    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:07.185046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.185016    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"]
Oct 30 14:30:07.192115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.192094    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:30:07.255708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.255673    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n\" (UID: \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:07.255847 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.255737    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqzs\" (UniqueName: \"kubernetes.io/projected/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-kube-api-access-8xqzs\") pod \"maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n\" (UID: \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:07.255953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.255930    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n\" (UID: \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:07.357515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.357482    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n\" (UID: \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:07.357709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.357583    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqzs\" (UniqueName: \"kubernetes.io/projected/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-kube-api-access-8xqzs\") pod \"maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n\" (UID: \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:07.357813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.357795    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n\" (UID: \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:07.358245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.358215    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n\" (UID: \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:07.360093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.360075    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n\" (UID: \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:07.384101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.384079    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqzs\" (UniqueName: \"kubernetes.io/projected/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-kube-api-access-8xqzs\") pod \"maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n\" (UID: \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:07.483686 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.483593    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:07.852643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:07.852585    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"]
Oct 30 14:30:07.853049 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:30:07.853011    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b2ffdc_31e7_4b6f_88fe_91ab106c4eb0.slice/crio-1bbba6b6b1ba3f09788e5b9a426ce50b087061179772bd49830061781fbbdc33 WatchSource:0}: Error finding container 1bbba6b6b1ba3f09788e5b9a426ce50b087061179772bd49830061781fbbdc33: Status 404 returned error can't find the container with id 1bbba6b6b1ba3f09788e5b9a426ce50b087061179772bd49830061781fbbdc33
Oct 30 14:30:08.566029 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:08.565993    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n" event={"ID":"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0","Type":"ContainerStarted","Data":"511741a7ce1ddda293b774d1c88438bac1a07b5248112bdfff08936a6ec595a0"}
Oct 30 14:30:08.566421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:08.566056    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n" event={"ID":"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0","Type":"ContainerStarted","Data":"1bbba6b6b1ba3f09788e5b9a426ce50b087061179772bd49830061781fbbdc33"}
Oct 30 14:30:08.617106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:08.617060    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n" podStartSLOduration=1.617047347 podStartE2EDuration="1.617047347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:08.613812693 +0000 UTC m=+2473.645711632" watchObservedRunningTime="2026-10-30 14:30:08.617047347 +0000 UTC m=+2473.648946285"
Oct 30 14:30:09.183596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:09.183196    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:30:09.575737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:09.575708    4530 generic.go:356] "Generic (PLEG): container finished" podID="26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0" containerID="511741a7ce1ddda293b774d1c88438bac1a07b5248112bdfff08936a6ec595a0" exitCode=0
Oct 30 14:30:09.576215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:09.575791    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n" event={"ID":"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0","Type":"ContainerDied","Data":"511741a7ce1ddda293b774d1c88438bac1a07b5248112bdfff08936a6ec595a0"}
Oct 30 14:30:10.075680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:10.075637    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r" podUID="8740a5a1-f59a-4de3-aa24-0e235518ebed" containerName="main" containerID="cri-o://b9be0d11ae5fd5fa04e567f892e42a4eb082f2fce95ae71ec9a14214488607da" gracePeriod=45
Oct 30 14:30:10.452142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:10.452037    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r" podUID="8740a5a1-f59a-4de3-aa24-0e235518ebed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.233:8000/ready\": dial tcp 10.133.0.233:8000: connect: connection refused"
Oct 30 14:30:10.622970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:10.622930    4530 generic.go:356] "Generic (PLEG): container finished" podID="8740a5a1-f59a-4de3-aa24-0e235518ebed" containerID="b9be0d11ae5fd5fa04e567f892e42a4eb082f2fce95ae71ec9a14214488607da" exitCode=0
Oct 30 14:30:10.623410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:10.623382    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r" event={"ID":"8740a5a1-f59a-4de3-aa24-0e235518ebed","Type":"ContainerDied","Data":"b9be0d11ae5fd5fa04e567f892e42a4eb082f2fce95ae71ec9a14214488607da"}
Oct 30 14:30:11.031616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.028488    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:30:11.067183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.064812    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:11.141159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.141103    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-tmp-dir\") pod \"8740a5a1-f59a-4de3-aa24-0e235518ebed\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") "
Oct 30 14:30:11.141449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.141419    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-home\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-home\") pod \"8740a5a1-f59a-4de3-aa24-0e235518ebed\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") "
Oct 30 14:30:11.141575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.141468    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-model-cache\") pod \"8740a5a1-f59a-4de3-aa24-0e235518ebed\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") "
Oct 30 14:30:11.141575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.141553    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8740a5a1-f59a-4de3-aa24-0e235518ebed-kube-api-access-4qr96\" (UniqueName: \"kubernetes.io/projected/8740a5a1-f59a-4de3-aa24-0e235518ebed-kube-api-access-4qr96\") pod \"8740a5a1-f59a-4de3-aa24-0e235518ebed\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") "
Oct 30 14:30:11.141764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.141616    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8740a5a1-f59a-4de3-aa24-0e235518ebed-tls-certs\" (UniqueName: \"kubernetes.io/secret/8740a5a1-f59a-4de3-aa24-0e235518ebed-tls-certs\") pod \"8740a5a1-f59a-4de3-aa24-0e235518ebed\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") "
Oct 30 14:30:11.141764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.141677    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-dshm\") pod \"8740a5a1-f59a-4de3-aa24-0e235518ebed\" (UID: \"8740a5a1-f59a-4de3-aa24-0e235518ebed\") "
Oct 30 14:30:11.143053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.142973    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-home" pod "8740a5a1-f59a-4de3-aa24-0e235518ebed" (UID: "8740a5a1-f59a-4de3-aa24-0e235518ebed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:11.143260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.143202    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-tmp-dir" pod "8740a5a1-f59a-4de3-aa24-0e235518ebed" (UID: "8740a5a1-f59a-4de3-aa24-0e235518ebed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:11.145043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.144933    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-model-cache" pod "8740a5a1-f59a-4de3-aa24-0e235518ebed" (UID: "8740a5a1-f59a-4de3-aa24-0e235518ebed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:11.159717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.159652    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8740a5a1-f59a-4de3-aa24-0e235518ebed-tls-certs" pod "8740a5a1-f59a-4de3-aa24-0e235518ebed" (UID: "8740a5a1-f59a-4de3-aa24-0e235518ebed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:30:11.252015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.245184    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-xks-ca\") pod \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\" (UID: \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\") "
Oct 30 14:30:11.252015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.245347    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-openshift-ca\") pod \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\" (UID: \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\") "
Oct 30 14:30:11.252015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.245406    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-kube-api-access-8xqzs\" (UniqueName: \"kubernetes.io/projected/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-kube-api-access-8xqzs\") pod \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\" (UID: \"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0\") "
Oct 30 14:30:11.252015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.246247    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:11.252015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.246275    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:11.252015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.246290    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8740a5a1-f59a-4de3-aa24-0e235518ebed-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:11.252015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.246326    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:11.267650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.267614    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-dshm" pod "8740a5a1-f59a-4de3-aa24-0e235518ebed" (UID: "8740a5a1-f59a-4de3-aa24-0e235518ebed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:11.269367 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.267856    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-openshift-ca" pod "26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0" (UID: "26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:30:11.269367 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.268920    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8740a5a1-f59a-4de3-aa24-0e235518ebed-kube-api-access-4qr96" pod "8740a5a1-f59a-4de3-aa24-0e235518ebed" (UID: "8740a5a1-f59a-4de3-aa24-0e235518ebed"). InnerVolumeSpecName "kube-api-access-4qr96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:30:11.273748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.273719    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-kube-api-access-8xqzs" pod "26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0" (UID: "26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0"). InnerVolumeSpecName "kube-api-access-8xqzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:30:11.276185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.276149    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-xks-ca" pod "26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0" (UID: "26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:30:11.348154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.348122    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qr96\" (UniqueName: \"kubernetes.io/projected/8740a5a1-f59a-4de3-aa24-0e235518ebed-kube-api-access-4qr96\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:11.348276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.348158    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8740a5a1-f59a-4de3-aa24-0e235518ebed-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:11.348276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.348175    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:11.348276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.348189    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:11.348276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.348202    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xqzs\" (UniqueName: \"kubernetes.io/projected/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0-kube-api-access-8xqzs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:11.372606 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:11.372565    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:30:11.632381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.632348    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"
Oct 30 14:30:11.632966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.632379    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r" event={"ID":"8740a5a1-f59a-4de3-aa24-0e235518ebed","Type":"ContainerDied","Data":"e0815b130cf860a662c88fffdd5583bf5e69d4cb7ef9b99aef592e33f26fd6b9"}
Oct 30 14:30:11.632966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.632427    4530 scope.go:122] "RemoveContainer" containerID="b9be0d11ae5fd5fa04e567f892e42a4eb082f2fce95ae71ec9a14214488607da"
Oct 30 14:30:11.634166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.634143    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n" event={"ID":"26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0","Type":"ContainerDied","Data":"1bbba6b6b1ba3f09788e5b9a426ce50b087061179772bd49830061781fbbdc33"}
Oct 30 14:30:11.634293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.634191    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bbba6b6b1ba3f09788e5b9a426ce50b087061179772bd49830061781fbbdc33"
Oct 30 14:30:11.634293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.634146    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"
Oct 30 14:30:11.752563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.752322    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"]
Oct 30 14:30:11.760229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:11.760062    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d15602/test-model-d15602-kserve-c9cfcd95f-w9t7r"]
Oct 30 14:30:13.119243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:13.119210    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8740a5a1-f59a-4de3-aa24-0e235518ebed" path="/var/lib/kubelet/pods/8740a5a1-f59a-4de3-aa24-0e235518ebed/volumes"
Oct 30 14:30:14.108175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:14.108135    4530 scope.go:122] "RemoveContainer" containerID="1ee1526f83190fb84706c23f5c58647e0a37fef2f384131536fb743e8094e482"
Oct 30 14:30:14.670055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:14.670009    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg" event={"ID":"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f","Type":"ContainerStarted","Data":"1ee7e590feadad41ba46f1ba2759f55b1ec682d19b2f4ef16d5cf02d271bf1a0"}
Oct 30 14:30:14.726920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:14.726869    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg" podStartSLOduration=14.726855629 podStartE2EDuration="14.726855629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:14.722898399 +0000 UTC m=+2479.754797338" watchObservedRunningTime="2026-10-30 14:30:14.726855629 +0000 UTC m=+2479.758754567"
Oct 30 14:30:15.702566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:15.701598    4530 generic.go:356] "Generic (PLEG): container finished" podID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" containerID="1ee7e590feadad41ba46f1ba2759f55b1ec682d19b2f4ef16d5cf02d271bf1a0" exitCode=60
Oct 30 14:30:15.702566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:15.701662    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg" event={"ID":"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f","Type":"ContainerDied","Data":"1ee7e590feadad41ba46f1ba2759f55b1ec682d19b2f4ef16d5cf02d271bf1a0"}
Oct 30 14:30:15.702566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:15.701801    4530 scope.go:122] "RemoveContainer" containerID="1ee1526f83190fb84706c23f5c58647e0a37fef2f384131536fb743e8094e482"
Oct 30 14:30:15.702566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:15.702308    4530 scope.go:122] "RemoveContainer" containerID="1ee7e590feadad41ba46f1ba2759f55b1ec682d19b2f4ef16d5cf02d271bf1a0"
Oct 30 14:30:15.703262 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:15.702740    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg_odh-ai-gateway-infra(53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f"
Oct 30 14:30:15.741780 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:15.741749    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"]
Oct 30 14:30:16.899507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:16.899482    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"
Oct 30 14:30:16.947640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:16.947610    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-kube-api-access-xmbrk\" (UniqueName: \"kubernetes.io/projected/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-kube-api-access-xmbrk\") pod \"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f\" (UID: \"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f\") "
Oct 30 14:30:16.955721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:16.947820    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-maas-api-service-ca\") pod \"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f\" (UID: \"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f\") "
Oct 30 14:30:16.955721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:16.950715    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-maas-api-service-ca" pod "53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" (UID: "53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:30:16.974582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:16.964235    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-kube-api-access-xmbrk" pod "53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" (UID: "53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f"). InnerVolumeSpecName "kube-api-access-xmbrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:30:17.049265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:17.049237    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmbrk\" (UniqueName: \"kubernetes.io/projected/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-kube-api-access-xmbrk\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:17.049265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:17.049264    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:17.107696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:17.107667    4530 scope.go:122] "RemoveContainer" containerID="7bf4559295f0a83d56eb1b4f3e4a87cf4d1aadc941ffac05f477d5a66ed4740d"
Oct 30 14:30:17.726875 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:17.726795    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"
Oct 30 14:30:17.726875 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:17.726834    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg" event={"ID":"53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f","Type":"ContainerDied","Data":"5fe2a0b6abf01a478e3503978cb644fdaa343a9808fe75781e758344705b2ab7"}
Oct 30 14:30:17.727102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:17.726894    4530 scope.go:122] "RemoveContainer" containerID="1ee7e590feadad41ba46f1ba2759f55b1ec682d19b2f4ef16d5cf02d271bf1a0"
Oct 30 14:30:17.729483 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:17.729458    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb" event={"ID":"b90d5d9a-2c23-4cdf-9317-13f9803ca44b","Type":"ContainerStarted","Data":"aea8bfffded7fc017f75f323249156e36f10c9c2aa5e25408fea1fb9f5fd9005"}
Oct 30 14:30:17.865532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:17.865495    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"]
Oct 30 14:30:17.870027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:17.869993    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1d3696-29889510-sj6jg"]
Oct 30 14:30:18.108356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:18.108331    4530 scope.go:122] "RemoveContainer" containerID="43b7d2219c6914092450420c2e6f9075b36172b0213ac562dca8bfc590866b2e"
Oct 30 14:30:18.742280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:18.742242    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss" event={"ID":"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b","Type":"ContainerStarted","Data":"436bc9808d7cb22b2846e0aa346886738845dcdde1be7efb6c020519f16ff1b3"}
Oct 30 14:30:18.745952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:18.745921    4530 generic.go:356] "Generic (PLEG): container finished" podID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" containerID="aea8bfffded7fc017f75f323249156e36f10c9c2aa5e25408fea1fb9f5fd9005" exitCode=60
Oct 30 14:30:18.746101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:18.745992    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb" event={"ID":"b90d5d9a-2c23-4cdf-9317-13f9803ca44b","Type":"ContainerDied","Data":"aea8bfffded7fc017f75f323249156e36f10c9c2aa5e25408fea1fb9f5fd9005"}
Oct 30 14:30:18.746101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:18.746057    4530 scope.go:122] "RemoveContainer" containerID="7bf4559295f0a83d56eb1b4f3e4a87cf4d1aadc941ffac05f477d5a66ed4740d"
Oct 30 14:30:18.746458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:18.746433    4530 scope.go:122] "RemoveContainer" containerID="aea8bfffded7fc017f75f323249156e36f10c9c2aa5e25408fea1fb9f5fd9005"
Oct 30 14:30:18.746849 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:18.746825    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb_odh-ai-gateway-infra(b90d5d9a-2c23-4cdf-9317-13f9803ca44b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b"
Oct 30 14:30:18.819243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:18.819216    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"]
Oct 30 14:30:19.117767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:19.117732    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" path="/var/lib/kubelet/pods/53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f/volumes"
Oct 30 14:30:19.760110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:19.760077    4530 generic.go:356] "Generic (PLEG): container finished" podID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" containerID="436bc9808d7cb22b2846e0aa346886738845dcdde1be7efb6c020519f16ff1b3" exitCode=60
Oct 30 14:30:19.760238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:19.760165    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss" event={"ID":"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b","Type":"ContainerDied","Data":"436bc9808d7cb22b2846e0aa346886738845dcdde1be7efb6c020519f16ff1b3"}
Oct 30 14:30:19.760299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:19.760251    4530 scope.go:122] "RemoveContainer" containerID="43b7d2219c6914092450420c2e6f9075b36172b0213ac562dca8bfc590866b2e"
Oct 30 14:30:19.760681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:19.760660    4530 scope.go:122] "RemoveContainer" containerID="436bc9808d7cb22b2846e0aa346886738845dcdde1be7efb6c020519f16ff1b3"
Oct 30 14:30:19.761089 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:19.761068    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss_odh-ai-gateway-infra(d645d1ed-30a7-4229-bc89-7bdf9fbeb19b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b"
Oct 30 14:30:19.809350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:19.809251    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"]
Oct 30 14:30:19.943284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:19.943257    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"
Oct 30 14:30:19.996771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:19.996743    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-maas-api-service-ca\") pod \"b90d5d9a-2c23-4cdf-9317-13f9803ca44b\" (UID: \"b90d5d9a-2c23-4cdf-9317-13f9803ca44b\") "
Oct 30 14:30:19.996963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:19.996806    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-kube-api-access-68qnt\" (UniqueName: \"kubernetes.io/projected/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-kube-api-access-68qnt\") pod \"b90d5d9a-2c23-4cdf-9317-13f9803ca44b\" (UID: \"b90d5d9a-2c23-4cdf-9317-13f9803ca44b\") "
Oct 30 14:30:19.998030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:19.997989    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-maas-api-service-ca" pod "b90d5d9a-2c23-4cdf-9317-13f9803ca44b" (UID: "b90d5d9a-2c23-4cdf-9317-13f9803ca44b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:30:20.000288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.000249    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-kube-api-access-68qnt" pod "b90d5d9a-2c23-4cdf-9317-13f9803ca44b" (UID: "b90d5d9a-2c23-4cdf-9317-13f9803ca44b"). InnerVolumeSpecName "kube-api-access-68qnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:30:20.098740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.098701    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:20.098740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.098738    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68qnt\" (UniqueName: \"kubernetes.io/projected/b90d5d9a-2c23-4cdf-9317-13f9803ca44b-kube-api-access-68qnt\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:20.772323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.772293    4530 generic.go:356] "Generic (PLEG): container finished" podID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerID="6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607" exitCode=0
Oct 30 14:30:20.772874 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.772359    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" event={"ID":"7a91e56c-d22c-40a4-a8f1-e144eece82d4","Type":"ContainerDied","Data":"6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607"}
Oct 30 14:30:20.772962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.772874    4530 scope.go:122] "RemoveContainer" containerID="6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607"
Oct 30 14:30:20.773800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.773777    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb" event={"ID":"b90d5d9a-2c23-4cdf-9317-13f9803ca44b","Type":"ContainerDied","Data":"af36c8a65085b513bdefe135e464a1d88a70f2d2fb90545039e9561d6ed91f05"}
Oct 30 14:30:20.773894 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.773801    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"
Oct 30 14:30:20.773894 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.773829    4530 scope.go:122] "RemoveContainer" containerID="aea8bfffded7fc017f75f323249156e36f10c9c2aa5e25408fea1fb9f5fd9005"
Oct 30 14:30:20.775144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.775108    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:30:20.775245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.775203    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:30:20.775305 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.775254    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:30:20.875983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.875950    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"]
Oct 30 14:30:20.881261 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.881232    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889510-l5jnb"]
Oct 30 14:30:20.999288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:20.999262    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"
Oct 30 14:30:21.014186 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.013800    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-kube-api-access-6q2hr\" (UniqueName: \"kubernetes.io/projected/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-kube-api-access-6q2hr\") pod \"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b\" (UID: \"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b\") "
Oct 30 14:30:21.014186 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.013924    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-maas-api-service-ca\") pod \"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b\" (UID: \"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b\") "
Oct 30 14:30:21.014446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.014276    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-maas-api-service-ca" pod "d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" (UID: "d645d1ed-30a7-4229-bc89-7bdf9fbeb19b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:30:21.014581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.014532    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:21.016073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.016048    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-kube-api-access-6q2hr" pod "d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" (UID: "d645d1ed-30a7-4229-bc89-7bdf9fbeb19b"). InnerVolumeSpecName "kube-api-access-6q2hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:30:21.115790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.115754    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q2hr\" (UniqueName: \"kubernetes.io/projected/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b-kube-api-access-6q2hr\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:21.121548 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.121463    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" path="/var/lib/kubelet/pods/b90d5d9a-2c23-4cdf-9317-13f9803ca44b/volumes"
Oct 30 14:30:21.202646 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:21.188601    4530 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd645d1ed_30a7_4229_bc89_7bdf9fbeb19b.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 14:30:21.800564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.798708    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" event={"ID":"7a91e56c-d22c-40a4-a8f1-e144eece82d4","Type":"ContainerStarted","Data":"52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e"}
Oct 30 14:30:21.810036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.808388    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss" event={"ID":"d645d1ed-30a7-4229-bc89-7bdf9fbeb19b","Type":"ContainerDied","Data":"f5128bbc8f2e8a3cc1ed6234bd5bbaa124f799ffab08b40346b0eb75fa2e90a3"}
Oct 30 14:30:21.810036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.808458    4530 scope.go:122] "RemoveContainer" containerID="436bc9808d7cb22b2846e0aa346886738845dcdde1be7efb6c020519f16ff1b3"
Oct 30 14:30:21.810036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.808617    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"
Oct 30 14:30:21.927030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.926991    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"]
Oct 30 14:30:21.937621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:21.937593    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889510-prxss"]
Oct 30 14:30:22.304876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:22.304841    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:30:22.305073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:22.305037    4530 patch_prober.go:29] interesting pod/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 30 14:30:22.305154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:22.305093    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 30 14:30:23.119723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:23.119690    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" path="/var/lib/kubelet/pods/d645d1ed-30a7-4229-bc89-7bdf9fbeb19b/volumes"
Oct 30 14:30:23.304932 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:23.304895    4530 patch_prober.go:29] interesting pod/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 30 14:30:23.305116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:23.304970    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 30 14:30:24.304952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.304911    4530 patch_prober.go:29] interesting pod/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 30 14:30:24.305401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.304984    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 30 14:30:24.516006 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.515972    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"]
Oct 30 14:30:24.516739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516720    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" containerName="cleanup"
Oct 30 14:30:24.516739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516739    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516762    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516768    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516781    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516786    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516793    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516798    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516808    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516813    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516829    4530 cpu_manager.go:414] "containerMap: removing container" podUID="26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0" containerName="revoke-keys"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516836    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0" containerName="revoke-keys"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516855    4530 cpu_manager.go:414] "containerMap: removing container" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516860    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516869    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8740a5a1-f59a-4de3-aa24-0e235518ebed" containerName="main"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516874    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8740a5a1-f59a-4de3-aa24-0e235518ebed" containerName="main"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516882    4530 cpu_manager.go:414] "containerMap: removing container" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516895    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516908    4530 cpu_manager.go:414] "containerMap: removing container" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" containerName="cleanup"
Oct 30 14:30:24.516925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.516914    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" containerName="cleanup"
Oct 30 14:30:24.517704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.517031    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" containerName="cleanup"
Oct 30 14:30:24.517704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.517042    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" containerName="cleanup"
Oct 30 14:30:24.517704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.517049    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" containerName="cleanup"
Oct 30 14:30:24.517704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.517057    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0" containerName="revoke-keys"
Oct 30 14:30:24.517704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.517064    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b91f88-14b5-4321-bd83-5e752955ae3b" containerName="istio-proxy"
Oct 30 14:30:24.517704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.517072    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" containerName="cleanup"
Oct 30 14:30:24.517704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.517077    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8740a5a1-f59a-4de3-aa24-0e235518ebed" containerName="main"
Oct 30 14:30:24.517704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.517082    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" containerName="cleanup"
Oct 30 14:30:24.517704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.517089    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" containerName="cleanup"
Oct 30 14:30:24.521296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.521271    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.537375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.537346    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"]
Oct 30 14:30:24.542891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.542861    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-019e5b\"/\"kube-root-ca.crt\""
Oct 30 14:30:24.555993 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.555935    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-019e5b\"/\"openshift-service-ca.crt\""
Oct 30 14:30:24.556095 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.556009    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-019e5b\"/\"default-dockercfg-w7pjt\""
Oct 30 14:30:24.556148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.556137    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-019e5b\"/\"test-model-019e5b-kserve-self-signed-certs\""
Oct 30 14:30:24.604649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.604502    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmnx\" (UniqueName: \"kubernetes.io/projected/47456e25-a0e3-4610-b455-4dd8f428ebf8-kube-api-access-xrmnx\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.604649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.604588    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-dshm\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.604902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.604754    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-home\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.604971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.604901    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47456e25-a0e3-4610-b455-4dd8f428ebf8-tls-certs\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.605040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.604991    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-tmp-dir\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.605040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.605021    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-model-cache\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.707180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.707143    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmnx\" (UniqueName: \"kubernetes.io/projected/47456e25-a0e3-4610-b455-4dd8f428ebf8-kube-api-access-xrmnx\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.707349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.707189    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-dshm\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.707349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.707289    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-home\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.707484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.707441    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47456e25-a0e3-4610-b455-4dd8f428ebf8-tls-certs\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.707586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.707557    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-tmp-dir\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.707650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.707591    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-model-cache\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.707836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.707812    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-home\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.707993 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.707970    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-tmp-dir\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.708090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.708063    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-model-cache\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.710358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.710333    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-dshm\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.710595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.710576    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47456e25-a0e3-4610-b455-4dd8f428ebf8-tls-certs\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.759432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.759358    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmnx\" (UniqueName: \"kubernetes.io/projected/47456e25-a0e3-4610-b455-4dd8f428ebf8-kube-api-access-xrmnx\") pod \"test-model-019e5b-kserve-64b9ff6587-2tb5w\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") " pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:24.880615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:24.877486    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:25.230019 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:30:25.229992    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47456e25_a0e3_4610_b455_4dd8f428ebf8.slice/crio-d9a29888b9d83c43eb12f9479e2085b469c496ea2001138e56c18b680f01f2bc WatchSource:0}: Error finding container d9a29888b9d83c43eb12f9479e2085b469c496ea2001138e56c18b680f01f2bc: Status 404 returned error can't find the container with id d9a29888b9d83c43eb12f9479e2085b469c496ea2001138e56c18b680f01f2bc
Oct 30 14:30:25.232010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:25.231826    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"]
Oct 30 14:30:25.305407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:25.305013    4530 patch_prober.go:29] interesting pod/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 30 14:30:25.305407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:25.305059    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 30 14:30:25.909086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:25.909050    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w" event={"ID":"47456e25-a0e3-4610-b455-4dd8f428ebf8","Type":"ContainerStarted","Data":"763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf"}
Oct 30 14:30:25.909086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:25.909094    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w" event={"ID":"47456e25-a0e3-4610-b455-4dd8f428ebf8","Type":"ContainerStarted","Data":"d9a29888b9d83c43eb12f9479e2085b469c496ea2001138e56c18b680f01f2bc"}
Oct 30 14:30:25.959652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:25.959598    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w" podStartSLOduration=1.959585036 podStartE2EDuration="1.959585036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:25.958605601 +0000 UTC m=+2490.990504546" watchObservedRunningTime="2026-10-30 14:30:25.959585036 +0000 UTC m=+2490.991483974"
Oct 30 14:30:26.305355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:26.305318    4530 patch_prober.go:29] interesting pod/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 30 14:30:26.305581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:26.305386    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 30 14:30:26.893565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:26.885630    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:27.304023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:27.303987    4530 patch_prober.go:29] interesting pod/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 30 14:30:27.304205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:27.304053    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 30 14:30:28.308433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:28.308403    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:30:28.308958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:28.308942    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:30:28.309692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:28.309666    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:30:29.266851 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.266816    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"]
Oct 30 14:30:29.267536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.267520    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" containerName="cleanup"
Oct 30 14:30:29.267603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.267551    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" containerName="cleanup"
Oct 30 14:30:29.267673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.267663    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90d5d9a-2c23-4cdf-9317-13f9803ca44b" containerName="cleanup"
Oct 30 14:30:29.267711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.267681    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a6c9f1-8d5c-4c57-921f-1fe2b65cdc9f" containerName="cleanup"
Oct 30 14:30:29.267711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.267692    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d645d1ed-30a7-4229-bc89-7bdf9fbeb19b" containerName="cleanup"
Oct 30 14:30:29.271583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.271536    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.290520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.290488    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"]
Oct 30 14:30:29.309817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.309792    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-1d6347-serving-cert\""
Oct 30 14:30:29.389259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.389225    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrj9x\" (UniqueName: \"kubernetes.io/projected/697e817c-3d63-4813-8d5b-84ed291e3f3f-kube-api-access-lrj9x\") pod \"maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.389428 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.389356    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.389428 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.389388    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.389650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.389625    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/697e817c-3d63-4813-8d5b-84ed291e3f3f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.491131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.491097    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/697e817c-3d63-4813-8d5b-84ed291e3f3f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.491978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.491938    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/697e817c-3d63-4813-8d5b-84ed291e3f3f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.492351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.492320    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrj9x\" (UniqueName: \"kubernetes.io/projected/697e817c-3d63-4813-8d5b-84ed291e3f3f-kube-api-access-lrj9x\") pod \"maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.492905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.492863    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.493082 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.492956    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.498410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.498385    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.546176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.509601    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.546176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.536330    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrj9x\" (UniqueName: \"kubernetes.io/projected/697e817c-3d63-4813-8d5b-84ed291e3f3f-kube-api-access-lrj9x\") pod \"maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.587744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.587707    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:29.901310 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:30:29.901284    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697e817c_3d63_4813_8d5b_84ed291e3f3f.slice/crio-45dbd7a0eab676616c9bfa25d4f6047706bf255ff214e7cfec57081ad28244a7 WatchSource:0}: Error finding container 45dbd7a0eab676616c9bfa25d4f6047706bf255ff214e7cfec57081ad28244a7: Status 404 returned error can't find the container with id 45dbd7a0eab676616c9bfa25d4f6047706bf255ff214e7cfec57081ad28244a7
Oct 30 14:30:29.902831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.902803    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"]
Oct 30 14:30:29.902926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.902917    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:30:29.941909 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:29.941882    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs" event={"ID":"697e817c-3d63-4813-8d5b-84ed291e3f3f","Type":"ContainerStarted","Data":"45dbd7a0eab676616c9bfa25d4f6047706bf255ff214e7cfec57081ad28244a7"}
Oct 30 14:30:30.952121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:30.952079    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs" event={"ID":"697e817c-3d63-4813-8d5b-84ed291e3f3f","Type":"ContainerStarted","Data":"56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa"}
Oct 30 14:30:30.952581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:30.952185    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:31.001059 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:31.001012    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs" podStartSLOduration=1.599443093 podStartE2EDuration="2.000998576s" totalImagesPullingTime="401.555483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:30.998830864 +0000 UTC m=+2496.030729803" watchObservedRunningTime="2026-10-30 14:30:31.000998576 +0000 UTC m=+2496.032897513"
Oct 30 14:30:32.118436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:32.118400    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"]
Oct 30 14:30:34.879322 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:34.879288    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:34.892571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:34.892514    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:34.993995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:34.993965    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:30:36.966164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:36.966135    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:30:41.369746 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:41.369706    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:30:44.597156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.597119    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"]
Oct 30 14:30:44.601927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.601903    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.624182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.624150    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"]
Oct 30 14:30:44.630271 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.630246    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1d3696\"/\"openshift-service-ca.crt\""
Oct 30 14:30:44.630459 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.630345    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1d3696\"/\"kube-root-ca.crt\""
Oct 30 14:30:44.639005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.638981    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1d3696\"/\"default-dockercfg-4l2p8\""
Oct 30 14:30:44.639005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.639005    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1d3696\"/\"test-model-1d3696-kserve-self-signed-certs\""
Oct 30 14:30:44.677041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.677004    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-model-cache\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.677211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.677073    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-dshm\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.677266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.677228    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-tmp-dir\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.677318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.677277    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e85c33-012d-4136-abf2-029fb2600db2-tls-certs\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.677429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.677405    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-home\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.677506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.677467    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsg65\" (UniqueName: \"kubernetes.io/projected/f0e85c33-012d-4136-abf2-029fb2600db2-kube-api-access-hsg65\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.779336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.779298    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-model-cache\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.779522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.779342    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-dshm\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.779522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.779471    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-tmp-dir\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.779637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.779524    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e85c33-012d-4136-abf2-029fb2600db2-tls-certs\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.779698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.779635    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-home\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.779698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.779685    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsg65\" (UniqueName: \"kubernetes.io/projected/f0e85c33-012d-4136-abf2-029fb2600db2-kube-api-access-hsg65\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.779810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.779760    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-model-cache\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.779909 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.779890    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-tmp-dir\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.780029 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.780000    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-home\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.781748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.781717    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-dshm\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.782190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.782168    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e85c33-012d-4136-abf2-029fb2600db2-tls-certs\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.806190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.806163    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsg65\" (UniqueName: \"kubernetes.io/projected/f0e85c33-012d-4136-abf2-029fb2600db2-kube-api-access-hsg65\") pod \"test-model-1d3696-kserve-7fb699d56f-jmqjb\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") " pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:44.929363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:44.929249    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:45.260624 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:30:45.260593    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0e85c33_012d_4136_abf2_029fb2600db2.slice/crio-1129ce7007def886edf1969a478259baa364dd62ea50dacb2d83037288eb6b9c WatchSource:0}: Error finding container 1129ce7007def886edf1969a478259baa364dd62ea50dacb2d83037288eb6b9c: Status 404 returned error can't find the container with id 1129ce7007def886edf1969a478259baa364dd62ea50dacb2d83037288eb6b9c
Oct 30 14:30:45.262446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:45.262375    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"]
Oct 30 14:30:46.075514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:46.075478    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb" event={"ID":"f0e85c33-012d-4136-abf2-029fb2600db2","Type":"ContainerStarted","Data":"1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232"}
Oct 30 14:30:46.075935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:46.075523    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb" event={"ID":"f0e85c33-012d-4136-abf2-029fb2600db2","Type":"ContainerStarted","Data":"1129ce7007def886edf1969a478259baa364dd62ea50dacb2d83037288eb6b9c"}
Oct 30 14:30:46.124798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:46.124630    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb" podStartSLOduration=2.1246153740000002 podStartE2EDuration="2.124615374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:46.122255673 +0000 UTC m=+2511.154154593" watchObservedRunningTime="2026-10-30 14:30:46.124615374 +0000 UTC m=+2511.156514312"
Oct 30 14:30:46.930791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:46.930755    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:47.140344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.140307    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m" podUID="6be6f5b8-5567-4c76-96be-dcdc904c4f37" containerName="main" containerID="cri-o://8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e" gracePeriod=45
Oct 30 14:30:47.436924 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.436895    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:30:47.526611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.526517    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tmp-dir\") pod \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") "
Oct 30 14:30:47.526787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.526684    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-dshm\") pod \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") "
Oct 30 14:30:47.526787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.526740    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tls-certs\" (UniqueName: \"kubernetes.io/secret/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tls-certs\") pod \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") "
Oct 30 14:30:47.526933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.526783    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-model-cache\") pod \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") "
Oct 30 14:30:47.526933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.526806    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tmp-dir" pod "6be6f5b8-5567-4c76-96be-dcdc904c4f37" (UID: "6be6f5b8-5567-4c76-96be-dcdc904c4f37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:47.526933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.526819    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-home\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-home\") pod \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") "
Oct 30 14:30:47.527107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.527001    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-home" pod "6be6f5b8-5567-4c76-96be-dcdc904c4f37" (UID: "6be6f5b8-5567-4c76-96be-dcdc904c4f37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:47.527107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.527065    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-model-cache" pod "6be6f5b8-5567-4c76-96be-dcdc904c4f37" (UID: "6be6f5b8-5567-4c76-96be-dcdc904c4f37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:47.527223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.527113    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be6f5b8-5567-4c76-96be-dcdc904c4f37-kube-api-access-v2jzn\" (UniqueName: \"kubernetes.io/projected/6be6f5b8-5567-4c76-96be-dcdc904c4f37-kube-api-access-v2jzn\") pod \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\" (UID: \"6be6f5b8-5567-4c76-96be-dcdc904c4f37\") "
Oct 30 14:30:47.527911 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.527882    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:47.527911 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.527913    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:47.528125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.527926    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:47.529606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.529530    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-dshm" pod "6be6f5b8-5567-4c76-96be-dcdc904c4f37" (UID: "6be6f5b8-5567-4c76-96be-dcdc904c4f37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:30:47.529831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.529803    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be6f5b8-5567-4c76-96be-dcdc904c4f37-kube-api-access-v2jzn" pod "6be6f5b8-5567-4c76-96be-dcdc904c4f37" (UID: "6be6f5b8-5567-4c76-96be-dcdc904c4f37"). InnerVolumeSpecName "kube-api-access-v2jzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:30:47.529986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.529964    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tls-certs" pod "6be6f5b8-5567-4c76-96be-dcdc904c4f37" (UID: "6be6f5b8-5567-4c76-96be-dcdc904c4f37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:30:47.629859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.629827    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be6f5b8-5567-4c76-96be-dcdc904c4f37-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:47.629859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.629856    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2jzn\" (UniqueName: \"kubernetes.io/projected/6be6f5b8-5567-4c76-96be-dcdc904c4f37-kube-api-access-v2jzn\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:47.630062 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:47.629867    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be6f5b8-5567-4c76-96be-dcdc904c4f37-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:48.093535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:48.093498    4530 generic.go:356] "Generic (PLEG): container finished" podID="6be6f5b8-5567-4c76-96be-dcdc904c4f37" containerID="8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e" exitCode=0
Oct 30 14:30:48.093785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:48.093586    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"
Oct 30 14:30:48.093785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:48.093600    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m" event={"ID":"6be6f5b8-5567-4c76-96be-dcdc904c4f37","Type":"ContainerDied","Data":"8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e"}
Oct 30 14:30:48.093785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:48.093645    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m" event={"ID":"6be6f5b8-5567-4c76-96be-dcdc904c4f37","Type":"ContainerDied","Data":"840ddf1205b362e29c8ebae57992839563043ad2c115d0cff5c46204bdeecd33"}
Oct 30 14:30:48.093785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:48.093668    4530 scope.go:122] "RemoveContainer" containerID="8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e"
Oct 30 14:30:48.106159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:48.106141    4530 scope.go:122] "RemoveContainer" containerID="8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e"
Oct 30 14:30:48.106417 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:48.106396    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e\": container with ID starting with 8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e not found: ID does not exist" containerID="8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e"
Oct 30 14:30:48.106466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:48.106424    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e"} err="failed to get container status \"8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e\": rpc error: code = NotFound desc = could not find container \"8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e\": container with ID starting with 8a50610cfa9a9c4f85acfae2c2ec2a9d079144cbf9277b37a24aeec1db02fa9e not found: ID does not exist"
Oct 30 14:30:48.155727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:48.155700    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"]
Oct 30 14:30:48.169887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:48.169863    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-93549e/test-model-93549e-kserve-7c769d45f7-j6s5m"]
Oct 30 14:30:49.116005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:49.115968    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be6f5b8-5567-4c76-96be-dcdc904c4f37" path="/var/lib/kubelet/pods/6be6f5b8-5567-4c76-96be-dcdc904c4f37/volumes"
Oct 30 14:30:50.001852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.001821    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"]
Oct 30 14:30:50.002326 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.002018    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n" podUID="1bc93e64-3e9d-4e01-a0a8-27488bc4d312" containerName="maas-api" containerID="cri-o://b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477" gracePeriod=30
Oct 30 14:30:50.370923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.370495    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:30:50.536615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.529196    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-openshift-service-ca\") pod \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") "
Oct 30 14:30:50.536615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.529388    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-metrics-certs\") pod \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") "
Oct 30 14:30:50.571673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.571275    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-openshift-service-ca" pod "1bc93e64-3e9d-4e01-a0a8-27488bc4d312" (UID: "1bc93e64-3e9d-4e01-a0a8-27488bc4d312"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:30:50.574985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.574948    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-kube-api-access-8rmzv\" (UniqueName: \"kubernetes.io/projected/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-kube-api-access-8rmzv\") pod \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") "
Oct 30 14:30:50.575146 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.575010    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-maas-api-tls\") pod \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\" (UID: \"1bc93e64-3e9d-4e01-a0a8-27488bc4d312\") "
Oct 30 14:30:50.576738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.576663    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:50.583247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.579853    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-metrics-certs" pod "1bc93e64-3e9d-4e01-a0a8-27488bc4d312" (UID: "1bc93e64-3e9d-4e01-a0a8-27488bc4d312"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:30:50.586865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.586818    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-kube-api-access-8rmzv" pod "1bc93e64-3e9d-4e01-a0a8-27488bc4d312" (UID: "1bc93e64-3e9d-4e01-a0a8-27488bc4d312"). InnerVolumeSpecName "kube-api-access-8rmzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:30:50.588485 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.588446    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-maas-api-tls" pod "1bc93e64-3e9d-4e01-a0a8-27488bc4d312" (UID: "1bc93e64-3e9d-4e01-a0a8-27488bc4d312"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:30:50.678370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.678335    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rmzv\" (UniqueName: \"kubernetes.io/projected/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-kube-api-access-8rmzv\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:50.678370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.678366    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:50.678370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.678375    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc93e64-3e9d-4e01-a0a8-27488bc4d312-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:30:50.776602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.776532    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"]
Oct 30 14:30:50.781301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:50.781273    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9542c-dsp9n"]
Oct 30 14:30:51.118168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:51.118131    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0" path="/var/lib/kubelet/pods/26b2ffdc-31e7-4b6f-88fe-91ab106c4eb0/volumes"
Oct 30 14:30:51.124065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:51.124036    4530 generic.go:356] "Generic (PLEG): container finished" podID="1bc93e64-3e9d-4e01-a0a8-27488bc4d312" containerID="b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477" exitCode=0
Oct 30 14:30:51.124237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:51.124086    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"
Oct 30 14:30:51.124237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:51.124109    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n" event={"ID":"1bc93e64-3e9d-4e01-a0a8-27488bc4d312","Type":"ContainerDied","Data":"b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477"}
Oct 30 14:30:51.124237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:51.124174    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n" event={"ID":"1bc93e64-3e9d-4e01-a0a8-27488bc4d312","Type":"ContainerDied","Data":"94bd8179b50fe3b1ddafdedcd459ae4524c7c7d87c92cc5936f8fd4b4a364595"}
Oct 30 14:30:51.124237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:51.124207    4530 scope.go:122] "RemoveContainer" containerID="b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477"
Oct 30 14:30:51.157407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:51.157379    4530 scope.go:122] "RemoveContainer" containerID="b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477"
Oct 30 14:30:51.159111 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:30:51.159001    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477\": container with ID starting with b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477 not found: ID does not exist" containerID="b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477"
Oct 30 14:30:51.159111 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:51.159046    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477"} err="failed to get container status \"b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477\": rpc error: code = NotFound desc = could not find container \"b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477\": container with ID starting with b6e3d5264fa57803578b955dc8df63432298bffcca98e3ce34538931ed0e9477 not found: ID does not exist"
Oct 30 14:30:51.203397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:51.203367    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"]
Oct 30 14:30:51.216901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:51.216867    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a9542c-5b7986cc8-xwm4n"]
Oct 30 14:30:52.770165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:52.770128    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"]
Oct 30 14:30:53.116720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:53.116687    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc93e64-3e9d-4e01-a0a8-27488bc4d312" path="/var/lib/kubelet/pods/1bc93e64-3e9d-4e01-a0a8-27488bc4d312/volumes"
Oct 30 14:30:54.930254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:54.930217    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:54.942862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:54.942839    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:55.180475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:55.180398    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:30:57.770700 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:57.770652    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r" podUID="5e5ccec2-cb21-4dd9-94f0-6f39999b1d81" containerName="payload-processing" containerID="cri-o://2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb" gracePeriod=25
Oct 30 14:30:59.195599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:59.195567    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"]
Oct 30 14:30:59.196086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:30:59.195775    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy" containerID="cri-o://885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06" gracePeriod=30
Oct 30 14:31:00.253856 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:00.253823    4530 patch_prober.go:29] interesting pod/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:31:00.254325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:00.253880    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:31:00.480689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:00.480647    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:31:04.484997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.484973    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.579175    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-envoy\") pod \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") "
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.579374    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-credential-socket\") pod \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") "
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.579456    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-podinfo\") pod \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") "
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.579507    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-socket\") pod \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") "
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.579601    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-token\" (UniqueName: \"kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-token\") pod \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") "
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.579639    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-kube-api-access-rhbmp\" (UniqueName: \"kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-kube-api-access-rhbmp\") pod \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") "
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.579673    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-data\") pod \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") "
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.579725    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istiod-ca-cert\") pod \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") "
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.579941    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-certs\") pod \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\" (UID: \"28a281b3-e4e7-47ff-83b7-c34d7866ac19\") "
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.582278    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-certs" pod "28a281b3-e4e7-47ff-83b7-c34d7866ac19" (UID: "28a281b3-e4e7-47ff-83b7-c34d7866ac19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.587084    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-data" pod "28a281b3-e4e7-47ff-83b7-c34d7866ac19" (UID: "28a281b3-e4e7-47ff-83b7-c34d7866ac19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.587366    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istiod-ca-cert" pod "28a281b3-e4e7-47ff-83b7-c34d7866ac19" (UID: "28a281b3-e4e7-47ff-83b7-c34d7866ac19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.587731    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-socket" pod "28a281b3-e4e7-47ff-83b7-c34d7866ac19" (UID: "28a281b3-e4e7-47ff-83b7-c34d7866ac19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.588385    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-credential-socket" pod "28a281b3-e4e7-47ff-83b7-c34d7866ac19" (UID: "28a281b3-e4e7-47ff-83b7-c34d7866ac19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.589880    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-podinfo" pod "28a281b3-e4e7-47ff-83b7-c34d7866ac19" (UID: "28a281b3-e4e7-47ff-83b7-c34d7866ac19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:31:04.613784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.590343    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-envoy" pod "28a281b3-e4e7-47ff-83b7-c34d7866ac19" (UID: "28a281b3-e4e7-47ff-83b7-c34d7866ac19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:04.617155 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.617097    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-kube-api-access-rhbmp" pod "28a281b3-e4e7-47ff-83b7-c34d7866ac19" (UID: "28a281b3-e4e7-47ff-83b7-c34d7866ac19"). InnerVolumeSpecName "kube-api-access-rhbmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:31:04.622329 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.622273    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-token" pod "28a281b3-e4e7-47ff-83b7-c34d7866ac19" (UID: "28a281b3-e4e7-47ff-83b7-c34d7866ac19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:31:04.682182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.682151    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:04.682182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.682179    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:04.682347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.682190    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:04.682347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.682199    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:04.682347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.682209    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhbmp\" (UniqueName: \"kubernetes.io/projected/28a281b3-e4e7-47ff-83b7-c34d7866ac19-kube-api-access-rhbmp\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:04.682347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.682217    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:04.682347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.682225    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:04.682347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.682233    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:04.682347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:04.682241    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28a281b3-e4e7-47ff-83b7-c34d7866ac19-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:05.252372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:05.252335    4530 generic.go:356] "Generic (PLEG): container finished" podID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerID="885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06" exitCode=0
Oct 30 14:31:05.252558 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:05.252391    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"
Oct 30 14:31:05.252558 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:05.252434    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" event={"ID":"28a281b3-e4e7-47ff-83b7-c34d7866ac19","Type":"ContainerDied","Data":"885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06"}
Oct 30 14:31:05.252558 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:05.252495    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r" event={"ID":"28a281b3-e4e7-47ff-83b7-c34d7866ac19","Type":"ContainerDied","Data":"bacc46435d8fdffd753170979a6ea22f6dad13026b41d181370db317c8a7dc00"}
Oct 30 14:31:05.252558 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:05.252519    4530 scope.go:122] "RemoveContainer" containerID="885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06"
Oct 30 14:31:05.263573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:05.263554    4530 scope.go:122] "RemoveContainer" containerID="3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62"
Oct 30 14:31:05.273463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:05.273441    4530 scope.go:122] "RemoveContainer" containerID="885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06"
Oct 30 14:31:05.273735 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:31:05.273715    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06\": container with ID starting with 885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06 not found: ID does not exist" containerID="885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06"
Oct 30 14:31:05.273810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:05.273751    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06"} err="failed to get container status \"885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06\": rpc error: code = NotFound desc = could not find container \"885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06\": container with ID starting with 885bc64cd4320040b554fe0cc03b590f1fa1582146f335cf1b8c69d079262b06 not found: ID does not exist"
Oct 30 14:31:05.273810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:05.273775    4530 scope.go:122] "RemoveContainer" containerID="3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62"
Oct 30 14:31:05.274023 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:31:05.274004    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62\": container with ID starting with 3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62 not found: ID does not exist" containerID="3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62"
Oct 30 14:31:05.274074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:05.274028    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62"} err="failed to get container status \"3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62\": rpc error: code = NotFound desc = could not find container \"3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62\": container with ID starting with 3ed3d6f901e997c5b662a049cd175dedf4b927333e1f8f1fe16b529b8f482f62 not found: ID does not exist"
Oct 30 14:31:05.332085 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:05.332059    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"]
Oct 30 14:31:05.345722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:05.345696    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a9542c-openshift-default-8c5766f89-tsg6r"]
Oct 30 14:31:06.644573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:06.644519    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r" podUID="5e5ccec2-cb21-4dd9-94f0-6f39999b1d81" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:31:07.124861 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:07.124429    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" path="/var/lib/kubelet/pods/28a281b3-e4e7-47ff-83b7-c34d7866ac19/volumes"
Oct 30 14:31:11.368123 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:31:11.368070    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:31:16.644895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:16.644843    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r" podUID="5e5ccec2-cb21-4dd9-94f0-6f39999b1d81" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:31:18.061461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.061217    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"
Oct 30 14:31:18.070370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.070339    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"]
Oct 30 14:31:18.070615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.070584    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl" podUID="aa0da3cf-725c-4c81-9fff-0d0b202db949" containerName="maas-api" containerID="cri-o://6b080151439fad8352a933904c96831e11716e57fbf3098213c9eff02fb62f5c" gracePeriod=30
Oct 30 14:31:18.130714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.130684    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-plugins-config-volume\") pod \"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81\" (UID: \"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81\") "
Oct 30 14:31:18.130887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.130853    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-kube-api-access-7rqhf\" (UniqueName: \"kubernetes.io/projected/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-kube-api-access-7rqhf\") pod \"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81\" (UID: \"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81\") "
Oct 30 14:31:18.131164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.131133    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-plugins-config-volume" pod "5e5ccec2-cb21-4dd9-94f0-6f39999b1d81" (UID: "5e5ccec2-cb21-4dd9-94f0-6f39999b1d81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:31:18.131493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.131457    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:18.133274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.133246    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-kube-api-access-7rqhf" pod "5e5ccec2-cb21-4dd9-94f0-6f39999b1d81" (UID: "5e5ccec2-cb21-4dd9-94f0-6f39999b1d81"). InnerVolumeSpecName "kube-api-access-7rqhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:31:18.234277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.234051    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rqhf\" (UniqueName: \"kubernetes.io/projected/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81-kube-api-access-7rqhf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:18.424754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.424717    4530 generic.go:356] "Generic (PLEG): container finished" podID="aa0da3cf-725c-4c81-9fff-0d0b202db949" containerID="6b080151439fad8352a933904c96831e11716e57fbf3098213c9eff02fb62f5c" exitCode=0
Oct 30 14:31:18.424977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.424899    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl" event={"ID":"aa0da3cf-725c-4c81-9fff-0d0b202db949","Type":"ContainerDied","Data":"6b080151439fad8352a933904c96831e11716e57fbf3098213c9eff02fb62f5c"}
Oct 30 14:31:18.427177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.426317    4530 generic.go:356] "Generic (PLEG): container finished" podID="5e5ccec2-cb21-4dd9-94f0-6f39999b1d81" containerID="2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb" exitCode=0
Oct 30 14:31:18.427177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.426379    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r" event={"ID":"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81","Type":"ContainerDied","Data":"2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb"}
Oct 30 14:31:18.427177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.426418    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r" event={"ID":"5e5ccec2-cb21-4dd9-94f0-6f39999b1d81","Type":"ContainerDied","Data":"6adae44558e68628314200fe6db82fff387f7f22f6a7a6ef4409dc952f6d6b4f"}
Oct 30 14:31:18.427177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.426449    4530 scope.go:122] "RemoveContainer" containerID="2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb"
Oct 30 14:31:18.427177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.426633    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"
Oct 30 14:31:18.503507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.492204    4530 scope.go:122] "RemoveContainer" containerID="2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb"
Oct 30 14:31:18.503507 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:31:18.497447    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb\": container with ID starting with 2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb not found: ID does not exist" containerID="2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb"
Oct 30 14:31:18.503507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.497492    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb"} err="failed to get container status \"2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb\": rpc error: code = NotFound desc = could not find container \"2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb\": container with ID starting with 2431ecd1818216bf5fad7ff4c1ae61baabc4e309cde4cadfe3df43b1a0630fdb not found: ID does not exist"
Oct 30 14:31:18.546564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.525779    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:31:18.581040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.580942    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"]
Oct 30 14:31:18.589510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.589463    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d15602-5f9889cf64-z4t2r"]
Oct 30 14:31:18.658845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.655318    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-maas-api-tls\") pod \"aa0da3cf-725c-4c81-9fff-0d0b202db949\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") "
Oct 30 14:31:18.658845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.655372    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0da3cf-725c-4c81-9fff-0d0b202db949-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0da3cf-725c-4c81-9fff-0d0b202db949-openshift-service-ca\") pod \"aa0da3cf-725c-4c81-9fff-0d0b202db949\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") "
Oct 30 14:31:18.658845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.655415    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-metrics-certs\") pod \"aa0da3cf-725c-4c81-9fff-0d0b202db949\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") "
Oct 30 14:31:18.658845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.655477    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0da3cf-725c-4c81-9fff-0d0b202db949-kube-api-access-sm7t5\" (UniqueName: \"kubernetes.io/projected/aa0da3cf-725c-4c81-9fff-0d0b202db949-kube-api-access-sm7t5\") pod \"aa0da3cf-725c-4c81-9fff-0d0b202db949\" (UID: \"aa0da3cf-725c-4c81-9fff-0d0b202db949\") "
Oct 30 14:31:18.658845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.656993    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0da3cf-725c-4c81-9fff-0d0b202db949-openshift-service-ca" pod "aa0da3cf-725c-4c81-9fff-0d0b202db949" (UID: "aa0da3cf-725c-4c81-9fff-0d0b202db949"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:31:18.662627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.662533    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-metrics-certs" pod "aa0da3cf-725c-4c81-9fff-0d0b202db949" (UID: "aa0da3cf-725c-4c81-9fff-0d0b202db949"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:31:18.664095 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.664075    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0da3cf-725c-4c81-9fff-0d0b202db949-kube-api-access-sm7t5" pod "aa0da3cf-725c-4c81-9fff-0d0b202db949" (UID: "aa0da3cf-725c-4c81-9fff-0d0b202db949"). InnerVolumeSpecName "kube-api-access-sm7t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:31:18.664480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.664455    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-maas-api-tls" pod "aa0da3cf-725c-4c81-9fff-0d0b202db949" (UID: "aa0da3cf-725c-4c81-9fff-0d0b202db949"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:31:18.758124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.758019    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:18.758124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.758052    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0da3cf-725c-4c81-9fff-0d0b202db949-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:18.758124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.758066    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0da3cf-725c-4c81-9fff-0d0b202db949-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:18.758124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.758081    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm7t5\" (UniqueName: \"kubernetes.io/projected/aa0da3cf-725c-4c81-9fff-0d0b202db949-kube-api-access-sm7t5\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:18.961120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:18.961075    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"]
Oct 30 14:31:19.137294 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:19.137219    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5ccec2-cb21-4dd9-94f0-6f39999b1d81" path="/var/lib/kubelet/pods/5e5ccec2-cb21-4dd9-94f0-6f39999b1d81/volumes"
Oct 30 14:31:19.436791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:19.436710    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl" event={"ID":"aa0da3cf-725c-4c81-9fff-0d0b202db949","Type":"ContainerDied","Data":"d7e167bdb4adcb4acc414269d41ce8e17d00e004121f667c25c653b92626be57"}
Oct 30 14:31:19.436791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:19.436778    4530 scope.go:122] "RemoveContainer" containerID="6b080151439fad8352a933904c96831e11716e57fbf3098213c9eff02fb62f5c"
Oct 30 14:31:19.437016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:19.436718    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"
Oct 30 14:31:19.528852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:19.528815    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"]
Oct 30 14:31:19.545977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:19.545950    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d15602-57f9ddc875-g7rjl"]
Oct 30 14:31:21.116629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:21.116594    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0da3cf-725c-4c81-9fff-0d0b202db949" path="/var/lib/kubelet/pods/aa0da3cf-725c-4c81-9fff-0d0b202db949/volumes"
Oct 30 14:31:26.641304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:26.641269    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"]
Oct 30 14:31:26.643157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:26.643119    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy" containerID="cri-o://0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548" gracePeriod=30
Oct 30 14:31:29.663016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:29.662979    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"]
Oct 30 14:31:29.699172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:29.699147    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"]
Oct 30 14:31:30.647194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:30.647158    4530 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:31:30.647370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:30.647233    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:31:31.929764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:31.929735    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:31:32.086299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.086245    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-envoy\") pod \"89261fc0-4ce9-4d2a-b886-f730e9424523\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") "
Oct 30 14:31:32.086555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.086522    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-socket\") pod \"89261fc0-4ce9-4d2a-b886-f730e9424523\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") "
Oct 30 14:31:32.088688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.088663    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-data\") pod \"89261fc0-4ce9-4d2a-b886-f730e9424523\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") "
Oct 30 14:31:32.093451 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.089086    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-data" pod "89261fc0-4ce9-4d2a-b886-f730e9424523" (UID: "89261fc0-4ce9-4d2a-b886-f730e9424523"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:32.093736 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.089293    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-socket" pod "89261fc0-4ce9-4d2a-b886-f730e9424523" (UID: "89261fc0-4ce9-4d2a-b886-f730e9424523"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:32.094027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.094005    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-podinfo\") pod \"89261fc0-4ce9-4d2a-b886-f730e9424523\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") "
Oct 30 14:31:32.095367 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.095348    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-kube-api-access-zrth2\" (UniqueName: \"kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-kube-api-access-zrth2\") pod \"89261fc0-4ce9-4d2a-b886-f730e9424523\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") "
Oct 30 14:31:32.095876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.095848    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-certs\") pod \"89261fc0-4ce9-4d2a-b886-f730e9424523\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") "
Oct 30 14:31:32.096126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.096101    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89261fc0-4ce9-4d2a-b886-f730e9424523-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89261fc0-4ce9-4d2a-b886-f730e9424523-istiod-ca-cert\") pod \"89261fc0-4ce9-4d2a-b886-f730e9424523\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") "
Oct 30 14:31:32.096358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.096332    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-credential-socket\") pod \"89261fc0-4ce9-4d2a-b886-f730e9424523\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") "
Oct 30 14:31:32.096499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.096483    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-token\" (UniqueName: \"kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-token\") pod \"89261fc0-4ce9-4d2a-b886-f730e9424523\" (UID: \"89261fc0-4ce9-4d2a-b886-f730e9424523\") "
Oct 30 14:31:32.097333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.097299    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:32.097473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.097459    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:32.099239 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.098128    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89261fc0-4ce9-4d2a-b886-f730e9424523-istiod-ca-cert" pod "89261fc0-4ce9-4d2a-b886-f730e9424523" (UID: "89261fc0-4ce9-4d2a-b886-f730e9424523"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:31:32.099239 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.098364    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-certs" pod "89261fc0-4ce9-4d2a-b886-f730e9424523" (UID: "89261fc0-4ce9-4d2a-b886-f730e9424523"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:32.099239 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.098534    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-credential-socket" pod "89261fc0-4ce9-4d2a-b886-f730e9424523" (UID: "89261fc0-4ce9-4d2a-b886-f730e9424523"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:32.146328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.146175    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-podinfo" pod "89261fc0-4ce9-4d2a-b886-f730e9424523" (UID: "89261fc0-4ce9-4d2a-b886-f730e9424523"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:31:32.146328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.146288    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-envoy" pod "89261fc0-4ce9-4d2a-b886-f730e9424523" (UID: "89261fc0-4ce9-4d2a-b886-f730e9424523"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:32.157580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.157072    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-token" pod "89261fc0-4ce9-4d2a-b886-f730e9424523" (UID: "89261fc0-4ce9-4d2a-b886-f730e9424523"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:31:32.160595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.160558    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-kube-api-access-zrth2" pod "89261fc0-4ce9-4d2a-b886-f730e9424523" (UID: "89261fc0-4ce9-4d2a-b886-f730e9424523"). InnerVolumeSpecName "kube-api-access-zrth2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:31:32.202563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.199610    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:32.202563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.199640    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrth2\" (UniqueName: \"kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-kube-api-access-zrth2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:32.202563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.199651    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:32.202563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.199659    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89261fc0-4ce9-4d2a-b886-f730e9424523-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:32.202563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.199667    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:32.202563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.199675    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:32.202563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.199685    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89261fc0-4ce9-4d2a-b886-f730e9424523-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:32.541667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.541584    4530 generic.go:356] "Generic (PLEG): container finished" podID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerID="0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548" exitCode=0
Oct 30 14:31:32.541815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.541686    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"
Oct 30 14:31:32.541815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.541696    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" event={"ID":"89261fc0-4ce9-4d2a-b886-f730e9424523","Type":"ContainerDied","Data":"0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548"}
Oct 30 14:31:32.541815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.541766    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d" event={"ID":"89261fc0-4ce9-4d2a-b886-f730e9424523","Type":"ContainerDied","Data":"a5641656c2ee37abf4e8e742740a8cad8e39932948c25d91469bf416a4af592e"}
Oct 30 14:31:32.541815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.541779    4530 scope.go:122] "RemoveContainer" containerID="0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548"
Oct 30 14:31:32.553859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.553580    4530 scope.go:122] "RemoveContainer" containerID="559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69"
Oct 30 14:31:32.581527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.579939    4530 scope.go:122] "RemoveContainer" containerID="0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548"
Oct 30 14:31:32.581527 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:31:32.580988    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548\": container with ID starting with 0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548 not found: ID does not exist" containerID="0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548"
Oct 30 14:31:32.581527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.581034    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548"} err="failed to get container status \"0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548\": rpc error: code = NotFound desc = could not find container \"0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548\": container with ID starting with 0b97bc7cb9a97e542fb12e89f10491c75a1e0f531a80b7d58f9d1a6d7b3a5548 not found: ID does not exist"
Oct 30 14:31:32.581527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.581055    4530 scope.go:122] "RemoveContainer" containerID="559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69"
Oct 30 14:31:32.582667 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:31:32.582492    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69\": container with ID starting with 559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69 not found: ID does not exist" containerID="559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69"
Oct 30 14:31:32.582667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.582580    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69"} err="failed to get container status \"559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69\": rpc error: code = NotFound desc = could not find container \"559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69\": container with ID starting with 559e5fc612562f4fa80ad0ed2ef26d40cd22e4a10422b1083e83f04a2978ef69 not found: ID does not exist"
Oct 30 14:31:32.640564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.637604    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"]
Oct 30 14:31:32.643062 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:32.643029    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-cpt8d"]
Oct 30 14:31:33.116536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:33.116505    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" path="/var/lib/kubelet/pods/89261fc0-4ce9-4d2a-b886-f730e9424523/volumes"
Oct 30 14:31:33.990236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:33.990137    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq" podUID="f18d355c-edfe-4d27-bb1b-ff09d18fd43e" containerName="main" containerID="cri-o://e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3" gracePeriod=45
Oct 30 14:31:34.277035 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.277005    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:31:34.367411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.367371    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-home\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-home\") pod \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") "
Oct 30 14:31:34.367633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.367610    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tmp-dir\") pod \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") "
Oct 30 14:31:34.367692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.367678    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-dshm\") pod \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") "
Oct 30 14:31:34.367767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.367719    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-home" pod "f18d355c-edfe-4d27-bb1b-ff09d18fd43e" (UID: "f18d355c-edfe-4d27-bb1b-ff09d18fd43e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:34.367767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.367738    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tls-certs\") pod \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") "
Oct 30 14:31:34.367897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.367781    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-kube-api-access-9cgbt\" (UniqueName: \"kubernetes.io/projected/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-kube-api-access-9cgbt\") pod \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") "
Oct 30 14:31:34.367958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.367920    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tmp-dir" pod "f18d355c-edfe-4d27-bb1b-ff09d18fd43e" (UID: "f18d355c-edfe-4d27-bb1b-ff09d18fd43e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:34.367958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.367940    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-model-cache\") pod \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\" (UID: \"f18d355c-edfe-4d27-bb1b-ff09d18fd43e\") "
Oct 30 14:31:34.368610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.368579    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:34.368610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.368600    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-model-cache" pod "f18d355c-edfe-4d27-bb1b-ff09d18fd43e" (UID: "f18d355c-edfe-4d27-bb1b-ff09d18fd43e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:34.368806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.368610    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:34.370735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.370709    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tls-certs" pod "f18d355c-edfe-4d27-bb1b-ff09d18fd43e" (UID: "f18d355c-edfe-4d27-bb1b-ff09d18fd43e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:31:34.370735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.370722    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-dshm" pod "f18d355c-edfe-4d27-bb1b-ff09d18fd43e" (UID: "f18d355c-edfe-4d27-bb1b-ff09d18fd43e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:34.370904 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.370714    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-kube-api-access-9cgbt" pod "f18d355c-edfe-4d27-bb1b-ff09d18fd43e" (UID: "f18d355c-edfe-4d27-bb1b-ff09d18fd43e"). InnerVolumeSpecName "kube-api-access-9cgbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:31:34.470086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.470053    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:34.470086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.470080    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:34.470086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.470091    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cgbt\" (UniqueName: \"kubernetes.io/projected/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-kube-api-access-9cgbt\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:34.470341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.470100    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18d355c-edfe-4d27-bb1b-ff09d18fd43e-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:34.558795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.558760    4530 generic.go:356] "Generic (PLEG): container finished" podID="f18d355c-edfe-4d27-bb1b-ff09d18fd43e" containerID="e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3" exitCode=0
Oct 30 14:31:34.558983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.558809    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"
Oct 30 14:31:34.558983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.558841    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq" event={"ID":"f18d355c-edfe-4d27-bb1b-ff09d18fd43e","Type":"ContainerDied","Data":"e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3"}
Oct 30 14:31:34.558983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.558886    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq" event={"ID":"f18d355c-edfe-4d27-bb1b-ff09d18fd43e","Type":"ContainerDied","Data":"61cbcab851e3110e6a72db020987b6c0a1fdc3935be35394697f6242547a04ce"}
Oct 30 14:31:34.558983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.558903    4530 scope.go:122] "RemoveContainer" containerID="e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3"
Oct 30 14:31:34.570327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.570308    4530 scope.go:122] "RemoveContainer" containerID="e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3"
Oct 30 14:31:34.570653 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:31:34.570634    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3\": container with ID starting with e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3 not found: ID does not exist" containerID="e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3"
Oct 30 14:31:34.570705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.570663    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3"} err="failed to get container status \"e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3\": rpc error: code = NotFound desc = could not find container \"e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3\": container with ID starting with e08800982ab6c38327c5b9cd10830fe0e883661a234e24f7c44ce3a03dde9ed3 not found: ID does not exist"
Oct 30 14:31:34.620627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.620596    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"]
Oct 30 14:31:34.626183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.626158    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fde75d/test-model-fde75d-kserve-64b99544b5-rptrq"]
Oct 30 14:31:34.664091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.664037    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp" podUID="841ad8f3-2367-45ab-b6ff-5af3b631cdad" containerName="payload-processing" containerID="cri-o://fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c" gracePeriod=25
Oct 30 14:31:34.700398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.700349    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv" podUID="fc1b9781-ff8a-4981-a7fb-99a91c1d47f6" containerName="payload-pre-processing" containerID="cri-o://301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28" gracePeriod=25
Oct 30 14:31:34.934469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:34.934438    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"
Oct 30 14:31:35.120166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.119956    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841ad8f3-2367-45ab-b6ff-5af3b631cdad-kube-api-access-k8vkf\" (UniqueName: \"kubernetes.io/projected/841ad8f3-2367-45ab-b6ff-5af3b631cdad-kube-api-access-k8vkf\") pod \"841ad8f3-2367-45ab-b6ff-5af3b631cdad\" (UID: \"841ad8f3-2367-45ab-b6ff-5af3b631cdad\") "
Oct 30 14:31:35.120166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.120029    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/841ad8f3-2367-45ab-b6ff-5af3b631cdad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841ad8f3-2367-45ab-b6ff-5af3b631cdad-plugins-config-volume\") pod \"841ad8f3-2367-45ab-b6ff-5af3b631cdad\" (UID: \"841ad8f3-2367-45ab-b6ff-5af3b631cdad\") "
Oct 30 14:31:35.123302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.121942    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841ad8f3-2367-45ab-b6ff-5af3b631cdad-plugins-config-volume" pod "841ad8f3-2367-45ab-b6ff-5af3b631cdad" (UID: "841ad8f3-2367-45ab-b6ff-5af3b631cdad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:31:35.125964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.125934    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841ad8f3-2367-45ab-b6ff-5af3b631cdad-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:35.128238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.128204    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841ad8f3-2367-45ab-b6ff-5af3b631cdad-kube-api-access-k8vkf" pod "841ad8f3-2367-45ab-b6ff-5af3b631cdad" (UID: "841ad8f3-2367-45ab-b6ff-5af3b631cdad"). InnerVolumeSpecName "kube-api-access-k8vkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:31:35.133024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.133000    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f18d355c-edfe-4d27-bb1b-ff09d18fd43e" path="/var/lib/kubelet/pods/f18d355c-edfe-4d27-bb1b-ff09d18fd43e/volumes"
Oct 30 14:31:35.227117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.227018    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8vkf\" (UniqueName: \"kubernetes.io/projected/841ad8f3-2367-45ab-b6ff-5af3b631cdad-kube-api-access-k8vkf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:35.569434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.569397    4530 generic.go:356] "Generic (PLEG): container finished" podID="841ad8f3-2367-45ab-b6ff-5af3b631cdad" containerID="fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c" exitCode=0
Oct 30 14:31:35.569972 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.569452    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"
Oct 30 14:31:35.569972 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.569479    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp" event={"ID":"841ad8f3-2367-45ab-b6ff-5af3b631cdad","Type":"ContainerDied","Data":"fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c"}
Oct 30 14:31:35.569972 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.569536    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp" event={"ID":"841ad8f3-2367-45ab-b6ff-5af3b631cdad","Type":"ContainerDied","Data":"44a68af6c65069c8f11ec4c215c23792c6eaf3ec8c2ee0be05b3cc7804ec2ab7"}
Oct 30 14:31:35.569972 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.569565    4530 scope.go:122] "RemoveContainer" containerID="fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c"
Oct 30 14:31:35.580739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.580706    4530 scope.go:122] "RemoveContainer" containerID="fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c"
Oct 30 14:31:35.581003 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:31:35.580984    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c\": container with ID starting with fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c not found: ID does not exist" containerID="fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c"
Oct 30 14:31:35.581079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.581015    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c"} err="failed to get container status \"fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c\": rpc error: code = NotFound desc = could not find container \"fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c\": container with ID starting with fc59cc0e9cc993c0970f25041b9db57d33e7460f493806e07d4a96fa2f70ee7c not found: ID does not exist"
Oct 30 14:31:35.632070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.632036    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"]
Oct 30 14:31:35.646052 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:35.646022    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34bcc5-66cfd9bc76-r5frp"]
Oct 30 14:31:37.117705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:37.117659    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="841ad8f3-2367-45ab-b6ff-5af3b631cdad" path="/var/lib/kubelet/pods/841ad8f3-2367-45ab-b6ff-5af3b631cdad/volumes"
Oct 30 14:31:38.715004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:38.714922    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv" podUID="fc1b9781-ff8a-4981-a7fb-99a91c1d47f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:31:41.373636 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:31:41.373590    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:31:44.233144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233110    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"]
Oct 30 14:31:44.233812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233798    4530 cpu_manager.go:414] "containerMap: removing container" podUID="5e5ccec2-cb21-4dd9-94f0-6f39999b1d81" containerName="payload-processing"
Oct 30 14:31:44.233812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233813    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5ccec2-cb21-4dd9-94f0-6f39999b1d81" containerName="payload-processing"
Oct 30 14:31:44.233906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233832    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6be6f5b8-5567-4c76-96be-dcdc904c4f37" containerName="main"
Oct 30 14:31:44.233906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233838    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be6f5b8-5567-4c76-96be-dcdc904c4f37" containerName="main"
Oct 30 14:31:44.233906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233847    4530 cpu_manager.go:414] "containerMap: removing container" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy"
Oct 30 14:31:44.233906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233853    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy"
Oct 30 14:31:44.233906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233863    4530 cpu_manager.go:414] "containerMap: removing container" podUID="841ad8f3-2367-45ab-b6ff-5af3b631cdad" containerName="payload-processing"
Oct 30 14:31:44.233906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233868    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841ad8f3-2367-45ab-b6ff-5af3b631cdad" containerName="payload-processing"
Oct 30 14:31:44.233906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233875    4530 cpu_manager.go:414] "containerMap: removing container" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy"
Oct 30 14:31:44.233906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233880    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy"
Oct 30 14:31:44.233906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233888    4530 cpu_manager.go:414] "containerMap: removing container" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy"
Oct 30 14:31:44.233906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233892    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy"
Oct 30 14:31:44.233906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233906    4530 cpu_manager.go:414] "containerMap: removing container" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy"
Oct 30 14:31:44.233906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233911    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233934    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1bc93e64-3e9d-4e01-a0a8-27488bc4d312" containerName="maas-api"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233939    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc93e64-3e9d-4e01-a0a8-27488bc4d312" containerName="maas-api"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233951    4530 cpu_manager.go:414] "containerMap: removing container" podUID="aa0da3cf-725c-4c81-9fff-0d0b202db949" containerName="maas-api"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233956    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0da3cf-725c-4c81-9fff-0d0b202db949" containerName="maas-api"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233967    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f18d355c-edfe-4d27-bb1b-ff09d18fd43e" containerName="main"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.233972    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f18d355c-edfe-4d27-bb1b-ff09d18fd43e" containerName="main"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.234086    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc93e64-3e9d-4e01-a0a8-27488bc4d312" containerName="maas-api"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.234096    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0da3cf-725c-4c81-9fff-0d0b202db949" containerName="maas-api"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.234104    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f18d355c-edfe-4d27-bb1b-ff09d18fd43e" containerName="main"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.234112    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5ccec2-cb21-4dd9-94f0-6f39999b1d81" containerName="payload-processing"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.234119    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be6f5b8-5567-4c76-96be-dcdc904c4f37" containerName="main"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.234126    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.234131    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="841ad8f3-2367-45ab-b6ff-5af3b631cdad" containerName="payload-processing"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.234137    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.234142    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="89261fc0-4ce9-4d2a-b886-f730e9424523" containerName="istio-proxy"
Oct 30 14:31:44.234303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.234150    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a281b3-e4e7-47ff-83b7-c34d7866ac19" containerName="istio-proxy"
Oct 30 14:31:44.239317 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.239297    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.257467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.257435    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"]
Oct 30 14:31:44.266076 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.266046    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-acefbb\"/\"kube-root-ca.crt\""
Oct 30 14:31:44.266240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.266223    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-acefbb\"/\"openshift-service-ca.crt\""
Oct 30 14:31:44.266329 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.266316    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-acefbb\"/\"auth-test-model-acefbb-kserve-self-signed-certs\""
Oct 30 14:31:44.271122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.271097    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-acefbb\"/\"default-dockercfg-2m6fj\""
Oct 30 14:31:44.289257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.289227    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhnm\" (UniqueName: \"kubernetes.io/projected/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-kube-api-access-klhnm\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.289427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.289280    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-model-cache\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.289427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.289339    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tls-certs\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.289427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.289390    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-home\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.289628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.289470    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tmp-dir\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.289628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.289529    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-dshm\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.391403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.391357    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhnm\" (UniqueName: \"kubernetes.io/projected/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-kube-api-access-klhnm\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.391642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.391425    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-model-cache\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.391642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.391457    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tls-certs\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.391642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.391498    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-home\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.391854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.391707    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tmp-dir\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.391854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.391774    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-dshm\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.391985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.391963    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-home\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.392045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.391998    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-model-cache\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.392094 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.392046    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tmp-dir\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.394037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.394010    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-dshm\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.394199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.394181    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tls-certs\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.418536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.418501    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhnm\" (UniqueName: \"kubernetes.io/projected/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-kube-api-access-klhnm\") pod \"auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") " pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:44.552977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:44.552948    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:45.053247 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:31:45.053215    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0c01eea_69d5_4c86_bc41_ccd1143c36e7.slice/crio-9382d9708e6005e8dd247e06bea8c6daf48c237eb66defe9ac2fffb2b3d2bc48 WatchSource:0}: Error finding container 9382d9708e6005e8dd247e06bea8c6daf48c237eb66defe9ac2fffb2b3d2bc48: Status 404 returned error can't find the container with id 9382d9708e6005e8dd247e06bea8c6daf48c237eb66defe9ac2fffb2b3d2bc48
Oct 30 14:31:45.054345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:45.054315    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"]
Oct 30 14:31:45.719285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:45.719251    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv" event={"ID":"e0c01eea-69d5-4c86-bc41-ccd1143c36e7","Type":"ContainerStarted","Data":"677633d62f6d0dee313a5b7ff71f2cc8956cdc2839d6d5e5ad0dc108b8063e75"}
Oct 30 14:31:45.719717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:45.719293    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv" event={"ID":"e0c01eea-69d5-4c86-bc41-ccd1143c36e7","Type":"ContainerStarted","Data":"9382d9708e6005e8dd247e06bea8c6daf48c237eb66defe9ac2fffb2b3d2bc48"}
Oct 30 14:31:45.765773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:45.765720    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv" podStartSLOduration=1.765703708 podStartE2EDuration="1.765703708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:31:45.765603799 +0000 UTC m=+2570.797502743" watchObservedRunningTime="2026-10-30 14:31:45.765703708 +0000 UTC m=+2570.797602647"
Oct 30 14:31:46.553289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:46.553253    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:48.714815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:48.714774    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv" podUID="fc1b9781-ff8a-4981-a7fb-99a91c1d47f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:31:52.719145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:52.719104    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"]
Oct 30 14:31:52.719656 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:52.719397    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy" containerID="cri-o://4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d" gracePeriod=30
Oct 30 14:31:54.144929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:54.144892    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"]
Oct 30 14:31:54.553380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:54.553346    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:54.566073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:54.566041    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:54.804593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:54.804478    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:31:55.015075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.015049    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"
Oct 30 14:31:55.106009 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.105931    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-kube-api-access-trl9z\" (UniqueName: \"kubernetes.io/projected/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-kube-api-access-trl9z\") pod \"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6\" (UID: \"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6\") "
Oct 30 14:31:55.106176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.106010    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-plugins-config-volume\") pod \"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6\" (UID: \"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6\") "
Oct 30 14:31:55.106340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.106317    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-plugins-config-volume" pod "fc1b9781-ff8a-4981-a7fb-99a91c1d47f6" (UID: "fc1b9781-ff8a-4981-a7fb-99a91c1d47f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:31:55.106533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.106519    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:55.108309 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.108286    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-kube-api-access-trl9z" pod "fc1b9781-ff8a-4981-a7fb-99a91c1d47f6" (UID: "fc1b9781-ff8a-4981-a7fb-99a91c1d47f6"). InnerVolumeSpecName "kube-api-access-trl9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:31:55.207996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.207960    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trl9z\" (UniqueName: \"kubernetes.io/projected/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6-kube-api-access-trl9z\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:55.341971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.341930    4530 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:31:55.342141 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.341997    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:31:55.800848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.800812    4530 generic.go:356] "Generic (PLEG): container finished" podID="fc1b9781-ff8a-4981-a7fb-99a91c1d47f6" containerID="301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28" exitCode=0
Oct 30 14:31:55.801077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.800870    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"
Oct 30 14:31:55.801077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.800907    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv" event={"ID":"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6","Type":"ContainerDied","Data":"301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28"}
Oct 30 14:31:55.801077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.800994    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv" event={"ID":"fc1b9781-ff8a-4981-a7fb-99a91c1d47f6","Type":"ContainerDied","Data":"bd4a0d7791973fae4a92b8a9e8874d20ac707bbc2d1ed08ce3ccc1761f05dda9"}
Oct 30 14:31:55.801077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.801024    4530 scope.go:122] "RemoveContainer" containerID="301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28"
Oct 30 14:31:55.813383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.813369    4530 scope.go:122] "RemoveContainer" containerID="301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28"
Oct 30 14:31:55.813708 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:31:55.813685    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28\": container with ID starting with 301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28 not found: ID does not exist" containerID="301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28"
Oct 30 14:31:55.813802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.813715    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28"} err="failed to get container status \"301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28\": rpc error: code = NotFound desc = could not find container \"301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28\": container with ID starting with 301868dddc8719cacc2eea2ed58ce19d22816dd5b5a6d995fd2b6f64febb7f28 not found: ID does not exist"
Oct 30 14:31:55.868521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.868482    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"]
Oct 30 14:31:55.878890 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:55.878860    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-34bcc5-556d888b55-7vxmv"]
Oct 30 14:31:56.226528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:56.226447    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:31:57.115856 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:57.115826    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1b9781-ff8a-4981-a7fb-99a91c1d47f6" path="/var/lib/kubelet/pods/fc1b9781-ff8a-4981-a7fb-99a91c1d47f6/volumes"
Oct 30 14:31:58.312176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.312153    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:31:58.335270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.335212    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-podinfo\") pod \"acdd5958-7f02-448b-8bcc-654433ec1cd2\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") "
Oct 30 14:31:58.335381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.335286    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-socket\") pod \"acdd5958-7f02-448b-8bcc-654433ec1cd2\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") "
Oct 30 14:31:58.335381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.335336    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-envoy\") pod \"acdd5958-7f02-448b-8bcc-654433ec1cd2\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") "
Oct 30 14:31:58.335381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.335355    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-token\" (UniqueName: \"kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-token\") pod \"acdd5958-7f02-448b-8bcc-654433ec1cd2\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") "
Oct 30 14:31:58.335557 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.335381    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-credential-socket\") pod \"acdd5958-7f02-448b-8bcc-654433ec1cd2\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") "
Oct 30 14:31:58.335557 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.335434    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-certs\") pod \"acdd5958-7f02-448b-8bcc-654433ec1cd2\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") "
Oct 30 14:31:58.335557 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.335524    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-data\") pod \"acdd5958-7f02-448b-8bcc-654433ec1cd2\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") "
Oct 30 14:31:58.335741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.335590    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acdd5958-7f02-448b-8bcc-654433ec1cd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acdd5958-7f02-448b-8bcc-654433ec1cd2-istiod-ca-cert\") pod \"acdd5958-7f02-448b-8bcc-654433ec1cd2\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") "
Oct 30 14:31:58.335741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.335627    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-kube-api-access-pgczq\" (UniqueName: \"kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-kube-api-access-pgczq\") pod \"acdd5958-7f02-448b-8bcc-654433ec1cd2\" (UID: \"acdd5958-7f02-448b-8bcc-654433ec1cd2\") "
Oct 30 14:31:58.335741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.335639    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-socket" pod "acdd5958-7f02-448b-8bcc-654433ec1cd2" (UID: "acdd5958-7f02-448b-8bcc-654433ec1cd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:58.336058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.335953    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-data" pod "acdd5958-7f02-448b-8bcc-654433ec1cd2" (UID: "acdd5958-7f02-448b-8bcc-654433ec1cd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:58.336058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.335969    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acdd5958-7f02-448b-8bcc-654433ec1cd2-istiod-ca-cert" pod "acdd5958-7f02-448b-8bcc-654433ec1cd2" (UID: "acdd5958-7f02-448b-8bcc-654433ec1cd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:31:58.336058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.336010    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-credential-socket" pod "acdd5958-7f02-448b-8bcc-654433ec1cd2" (UID: "acdd5958-7f02-448b-8bcc-654433ec1cd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:58.336358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.336114    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-certs" pod "acdd5958-7f02-448b-8bcc-654433ec1cd2" (UID: "acdd5958-7f02-448b-8bcc-654433ec1cd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:58.336358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.336287    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:58.336358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.336307    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:58.336358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.336322    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:58.336358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.336335    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:58.336358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.336347    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acdd5958-7f02-448b-8bcc-654433ec1cd2-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:58.337776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.337753    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-envoy" pod "acdd5958-7f02-448b-8bcc-654433ec1cd2" (UID: "acdd5958-7f02-448b-8bcc-654433ec1cd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:31:58.338039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.338015    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-kube-api-access-pgczq" pod "acdd5958-7f02-448b-8bcc-654433ec1cd2" (UID: "acdd5958-7f02-448b-8bcc-654433ec1cd2"). InnerVolumeSpecName "kube-api-access-pgczq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:31:58.338118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.338051    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-podinfo" pod "acdd5958-7f02-448b-8bcc-654433ec1cd2" (UID: "acdd5958-7f02-448b-8bcc-654433ec1cd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:31:58.338118 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.338069    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-token" pod "acdd5958-7f02-448b-8bcc-654433ec1cd2" (UID: "acdd5958-7f02-448b-8bcc-654433ec1cd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:31:58.437959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.437924    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgczq\" (UniqueName: \"kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-kube-api-access-pgczq\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:58.437959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.437957    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:58.438148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.437967    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:58.438148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.437977    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acdd5958-7f02-448b-8bcc-654433ec1cd2-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:31:58.936177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.936142    4530 generic.go:356] "Generic (PLEG): container finished" podID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerID="4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d" exitCode=0
Oct 30 14:31:58.936384 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.936193    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"
Oct 30 14:31:58.936384 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.936224    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" event={"ID":"acdd5958-7f02-448b-8bcc-654433ec1cd2","Type":"ContainerDied","Data":"4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d"}
Oct 30 14:31:58.936384 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.936280    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8" event={"ID":"acdd5958-7f02-448b-8bcc-654433ec1cd2","Type":"ContainerDied","Data":"42bbba1d277d81eac910f560cc8cfd3efa42ddcc42f853d7f582cae9bfaa89cd"}
Oct 30 14:31:58.936384 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.936309    4530 scope.go:122] "RemoveContainer" containerID="4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d"
Oct 30 14:31:58.948822 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.948694    4530 scope.go:122] "RemoveContainer" containerID="2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f"
Oct 30 14:31:58.961959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.961937    4530 scope.go:122] "RemoveContainer" containerID="4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d"
Oct 30 14:31:58.962265 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:31:58.962242    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d\": container with ID starting with 4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d not found: ID does not exist" containerID="4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d"
Oct 30 14:31:58.962360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.962277    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d"} err="failed to get container status \"4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d\": rpc error: code = NotFound desc = could not find container \"4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d\": container with ID starting with 4a49f5850c743a5a54934091c91ac37de5adbdc13f3d93546420fe8e24bbbc1d not found: ID does not exist"
Oct 30 14:31:58.962360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.962301    4530 scope.go:122] "RemoveContainer" containerID="2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f"
Oct 30 14:31:58.962623 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:31:58.962599    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f\": container with ID starting with 2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f not found: ID does not exist" containerID="2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f"
Oct 30 14:31:58.962677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:58.962636    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f"} err="failed to get container status \"2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f\": rpc error: code = NotFound desc = could not find container \"2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f\": container with ID starting with 2c8e0d630a3a925c04a81005b90d98c51719584b118d0de41630b14e0469cf1f not found: ID does not exist"
Oct 30 14:31:59.012641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:59.012616    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"]
Oct 30 14:31:59.026521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:59.026486    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-jx8k8"]
Oct 30 14:31:59.116094 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:31:59.116063    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" path="/var/lib/kubelet/pods/acdd5958-7f02-448b-8bcc-654433ec1cd2/volumes"
Oct 30 14:32:01.600601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.600567    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"]
Oct 30 14:32:01.601248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.601232    4530 cpu_manager.go:414] "containerMap: removing container" podUID="fc1b9781-ff8a-4981-a7fb-99a91c1d47f6" containerName="payload-pre-processing"
Oct 30 14:32:01.601302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.601249    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1b9781-ff8a-4981-a7fb-99a91c1d47f6" containerName="payload-pre-processing"
Oct 30 14:32:01.601302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.601268    4530 cpu_manager.go:414] "containerMap: removing container" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy"
Oct 30 14:32:01.601302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.601276    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy"
Oct 30 14:32:01.601413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.601345    4530 cpu_manager.go:414] "containerMap: removing container" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy"
Oct 30 14:32:01.601413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.601352    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy"
Oct 30 14:32:01.601524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.601477    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy"
Oct 30 14:32:01.601524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.601488    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1b9781-ff8a-4981-a7fb-99a91c1d47f6" containerName="payload-pre-processing"
Oct 30 14:32:01.606528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.606509    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.623867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.623840    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"]
Oct 30 14:32:01.633589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.633525    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-396d70-serving-cert\""
Oct 30 14:32:01.707127 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.707098    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.707310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.707248    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16cc6ed-d030-440f-97dc-49d67415754b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.707371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.707352    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.707521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.707498    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsw7z\" (UniqueName: \"kubernetes.io/projected/c16cc6ed-d030-440f-97dc-49d67415754b-kube-api-access-tsw7z\") pod \"maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.808938 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.808908    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16cc6ed-d030-440f-97dc-49d67415754b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.809098 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.809008    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.809164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.809152    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsw7z\" (UniqueName: \"kubernetes.io/projected/c16cc6ed-d030-440f-97dc-49d67415754b-kube-api-access-tsw7z\") pod \"maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.809221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.809200    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.809714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.809683    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16cc6ed-d030-440f-97dc-49d67415754b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.811639 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.811613    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.811777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.811713    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.860819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.860698    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsw7z\" (UniqueName: \"kubernetes.io/projected/c16cc6ed-d030-440f-97dc-49d67415754b-kube-api-access-tsw7z\") pod \"maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:01.920525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:01.920484    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:02.232181 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:32:02.232137    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16cc6ed_d030_440f_97dc_49d67415754b.slice/crio-8326013577aed472de9d4704bdfcf7b1428dd6be20a381846534a13e8b4d8f9b WatchSource:0}: Error finding container 8326013577aed472de9d4704bdfcf7b1428dd6be20a381846534a13e8b4d8f9b: Status 404 returned error can't find the container with id 8326013577aed472de9d4704bdfcf7b1428dd6be20a381846534a13e8b4d8f9b
Oct 30 14:32:02.233785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:02.233759    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"]
Oct 30 14:32:02.971366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:02.971271    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" event={"ID":"c16cc6ed-d030-440f-97dc-49d67415754b","Type":"ContainerStarted","Data":"b5c232842b69d0882f49b3fcb10ebc576eaf0f6d8739eb6cff28f70b8b02ae4d"}
Oct 30 14:32:02.971366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:02.971327    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" event={"ID":"c16cc6ed-d030-440f-97dc-49d67415754b","Type":"ContainerStarted","Data":"8326013577aed472de9d4704bdfcf7b1428dd6be20a381846534a13e8b4d8f9b"}
Oct 30 14:32:02.971366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:02.971358    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:03.047562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:03.030421    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" podStartSLOduration=1.625093935 podStartE2EDuration="2.030400763s" totalImagesPullingTime="405.306828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:32:03.029983862 +0000 UTC m=+2588.061882802" watchObservedRunningTime="2026-10-30 14:32:03.030400763 +0000 UTC m=+2588.062299704"
Oct 30 14:32:05.319700 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.319665    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"]
Oct 30 14:32:05.320457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.319898    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn" podUID="a381d859-fd85-4ec4-be51-3f6d6902cd01" containerName="maas-api" containerID="cri-o://ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24" gracePeriod=30
Oct 30 14:32:05.714792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.714761    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:32:05.931060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.930959    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a381d859-fd85-4ec4-be51-3f6d6902cd01-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a381d859-fd85-4ec4-be51-3f6d6902cd01-openshift-service-ca\") pod \"a381d859-fd85-4ec4-be51-3f6d6902cd01\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") "
Oct 30 14:32:05.931264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.931073    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-maas-api-tls\") pod \"a381d859-fd85-4ec4-be51-3f6d6902cd01\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") "
Oct 30 14:32:05.931264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.931156    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a381d859-fd85-4ec4-be51-3f6d6902cd01-kube-api-access-9wd7x\" (UniqueName: \"kubernetes.io/projected/a381d859-fd85-4ec4-be51-3f6d6902cd01-kube-api-access-9wd7x\") pod \"a381d859-fd85-4ec4-be51-3f6d6902cd01\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") "
Oct 30 14:32:05.931264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.931202    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-metrics-certs\") pod \"a381d859-fd85-4ec4-be51-3f6d6902cd01\" (UID: \"a381d859-fd85-4ec4-be51-3f6d6902cd01\") "
Oct 30 14:32:05.932357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.932238    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a381d859-fd85-4ec4-be51-3f6d6902cd01-openshift-service-ca" pod "a381d859-fd85-4ec4-be51-3f6d6902cd01" (UID: "a381d859-fd85-4ec4-be51-3f6d6902cd01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:32:05.939061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.939019    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-maas-api-tls" pod "a381d859-fd85-4ec4-be51-3f6d6902cd01" (UID: "a381d859-fd85-4ec4-be51-3f6d6902cd01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:32:05.939987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.939957    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a381d859-fd85-4ec4-be51-3f6d6902cd01-kube-api-access-9wd7x" pod "a381d859-fd85-4ec4-be51-3f6d6902cd01" (UID: "a381d859-fd85-4ec4-be51-3f6d6902cd01"). InnerVolumeSpecName "kube-api-access-9wd7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:32:05.941061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.941033    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-metrics-certs" pod "a381d859-fd85-4ec4-be51-3f6d6902cd01" (UID: "a381d859-fd85-4ec4-be51-3f6d6902cd01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:32:05.996095 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.996062    4530 generic.go:356] "Generic (PLEG): container finished" podID="a381d859-fd85-4ec4-be51-3f6d6902cd01" containerID="ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24" exitCode=0
Oct 30 14:32:05.996095 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.996119    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn" event={"ID":"a381d859-fd85-4ec4-be51-3f6d6902cd01","Type":"ContainerDied","Data":"ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24"}
Oct 30 14:32:05.996323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.996131    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"
Oct 30 14:32:05.996323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.996185    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn" event={"ID":"a381d859-fd85-4ec4-be51-3f6d6902cd01","Type":"ContainerDied","Data":"1b1636059fb49502e2c9a158117c4a607467cda4098cec1379630038c9a0e3af"}
Oct 30 14:32:05.996323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:05.996205    4530 scope.go:122] "RemoveContainer" containerID="ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24"
Oct 30 14:32:06.011987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:06.011966    4530 scope.go:122] "RemoveContainer" containerID="ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24"
Oct 30 14:32:06.012310 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:06.012292    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24\": container with ID starting with ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24 not found: ID does not exist" containerID="ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24"
Oct 30 14:32:06.012383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:06.012318    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24"} err="failed to get container status \"ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24\": rpc error: code = NotFound desc = could not find container \"ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24\": container with ID starting with ee6a43d82cb8f8bbc6cc4804524b7b9ba279c5e69042d77114fc3dde42090e24 not found: ID does not exist"
Oct 30 14:32:06.032858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:06.032832    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a381d859-fd85-4ec4-be51-3f6d6902cd01-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:06.032858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:06.032857    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:06.033014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:06.032866    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wd7x\" (UniqueName: \"kubernetes.io/projected/a381d859-fd85-4ec4-be51-3f6d6902cd01-kube-api-access-9wd7x\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:06.033014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:06.032875    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a381d859-fd85-4ec4-be51-3f6d6902cd01-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:06.057581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:06.057533    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"]
Oct 30 14:32:06.069275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:06.069253    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn"]
Oct 30 14:32:07.116190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:07.116154    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a381d859-fd85-4ec4-be51-3f6d6902cd01" path="/var/lib/kubelet/pods/a381d859-fd85-4ec4-be51-3f6d6902cd01/volumes"
Oct 30 14:32:08.646121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:08.646077    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-409364-5d8bc5656c-wt7rn" podUID="a381d859-fd85-4ec4-be51-3f6d6902cd01" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.224:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:32:08.980061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:08.979974    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.249:8443/health\": dial tcp 10.133.0.249:8443: connect: connection refused"
Oct 30 14:32:09.166862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.166818    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2" podUID="e02d6098-1f27-4b24-b0bc-49ff80d17724" containerName="main" containerID="cri-o://4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a" gracePeriod=45
Oct 30 14:32:09.452026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.452002    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:32:09.578698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.578665    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-tmp-dir\") pod \"e02d6098-1f27-4b24-b0bc-49ff80d17724\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") "
Oct 30 14:32:09.578873 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.578734    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-home\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-home\") pod \"e02d6098-1f27-4b24-b0bc-49ff80d17724\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") "
Oct 30 14:32:09.578934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.578905    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e02d6098-1f27-4b24-b0bc-49ff80d17724-tls-certs\" (UniqueName: \"kubernetes.io/secret/e02d6098-1f27-4b24-b0bc-49ff80d17724-tls-certs\") pod \"e02d6098-1f27-4b24-b0bc-49ff80d17724\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") "
Oct 30 14:32:09.578992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.578957    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-tmp-dir" pod "e02d6098-1f27-4b24-b0bc-49ff80d17724" (UID: "e02d6098-1f27-4b24-b0bc-49ff80d17724"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:09.579061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.578981    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-home" pod "e02d6098-1f27-4b24-b0bc-49ff80d17724" (UID: "e02d6098-1f27-4b24-b0bc-49ff80d17724"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:09.579061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.579025    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-model-cache\") pod \"e02d6098-1f27-4b24-b0bc-49ff80d17724\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") "
Oct 30 14:32:09.579061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.579052    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02d6098-1f27-4b24-b0bc-49ff80d17724-kube-api-access-9c79s\" (UniqueName: \"kubernetes.io/projected/e02d6098-1f27-4b24-b0bc-49ff80d17724-kube-api-access-9c79s\") pod \"e02d6098-1f27-4b24-b0bc-49ff80d17724\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") "
Oct 30 14:32:09.579228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.579116    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-dshm\") pod \"e02d6098-1f27-4b24-b0bc-49ff80d17724\" (UID: \"e02d6098-1f27-4b24-b0bc-49ff80d17724\") "
Oct 30 14:32:09.579228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.579215    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-model-cache" pod "e02d6098-1f27-4b24-b0bc-49ff80d17724" (UID: "e02d6098-1f27-4b24-b0bc-49ff80d17724"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:09.579716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.579689    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:09.579852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.579742    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:09.579852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.579756    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:09.581484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.581454    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e02d6098-1f27-4b24-b0bc-49ff80d17724-tls-certs" pod "e02d6098-1f27-4b24-b0bc-49ff80d17724" (UID: "e02d6098-1f27-4b24-b0bc-49ff80d17724"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:32:09.581484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.581465    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02d6098-1f27-4b24-b0bc-49ff80d17724-kube-api-access-9c79s" pod "e02d6098-1f27-4b24-b0bc-49ff80d17724" (UID: "e02d6098-1f27-4b24-b0bc-49ff80d17724"). InnerVolumeSpecName "kube-api-access-9c79s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:32:09.581665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.581532    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-dshm" pod "e02d6098-1f27-4b24-b0bc-49ff80d17724" (UID: "e02d6098-1f27-4b24-b0bc-49ff80d17724"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:09.680681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.680654    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02d6098-1f27-4b24-b0bc-49ff80d17724-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:09.680681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.680682    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c79s\" (UniqueName: \"kubernetes.io/projected/e02d6098-1f27-4b24-b0bc-49ff80d17724-kube-api-access-9c79s\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:09.681127 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:09.680698    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02d6098-1f27-4b24-b0bc-49ff80d17724-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:10.027740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:10.027653    4530 generic.go:356] "Generic (PLEG): container finished" podID="e02d6098-1f27-4b24-b0bc-49ff80d17724" containerID="4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a" exitCode=0
Oct 30 14:32:10.027954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:10.027923    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2" event={"ID":"e02d6098-1f27-4b24-b0bc-49ff80d17724","Type":"ContainerDied","Data":"4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a"}
Oct 30 14:32:10.028170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:10.028148    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2" event={"ID":"e02d6098-1f27-4b24-b0bc-49ff80d17724","Type":"ContainerDied","Data":"13e737ddb72996d06b10e50269fea15fdda795d2c70ef7ec6b9640ea5508e576"}
Oct 30 14:32:10.028251 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:10.028179    4530 scope.go:122] "RemoveContainer" containerID="4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a"
Oct 30 14:32:10.028328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:10.028317    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"
Oct 30 14:32:10.039454 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:10.039435    4530 scope.go:122] "RemoveContainer" containerID="4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a"
Oct 30 14:32:10.039803 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:10.039766    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a\": container with ID starting with 4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a not found: ID does not exist" containerID="4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a"
Oct 30 14:32:10.039882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:10.039801    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a"} err="failed to get container status \"4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a\": rpc error: code = NotFound desc = could not find container \"4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a\": container with ID starting with 4cf801f2eae4470ae6bb2251f941bd84cf4a796b335cffa676b72a99b760b61a not found: ID does not exist"
Oct 30 14:32:10.091090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:10.091060    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"]
Oct 30 14:32:10.105051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:10.105026    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f3c580/test-model-f3c580-kserve-54f79f67cc-rwbm2"]
Oct 30 14:32:11.115860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:11.115826    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e02d6098-1f27-4b24-b0bc-49ff80d17724" path="/var/lib/kubelet/pods/e02d6098-1f27-4b24-b0bc-49ff80d17724/volumes"
Oct 30 14:32:11.367799 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:11.367718    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:32:13.054589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:13.054558    4530 generic.go:356] "Generic (PLEG): container finished" podID="c16cc6ed-d030-440f-97dc-49d67415754b" containerID="b5c232842b69d0882f49b3fcb10ebc576eaf0f6d8739eb6cff28f70b8b02ae4d" exitCode=1
Oct 30 14:32:13.054589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:13.054574    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" event={"ID":"c16cc6ed-d030-440f-97dc-49d67415754b","Type":"ContainerDied","Data":"b5c232842b69d0882f49b3fcb10ebc576eaf0f6d8739eb6cff28f70b8b02ae4d"}
Oct 30 14:32:13.055026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:13.054985    4530 scope.go:122] "RemoveContainer" containerID="b5c232842b69d0882f49b3fcb10ebc576eaf0f6d8739eb6cff28f70b8b02ae4d"
Oct 30 14:32:14.063457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:14.063421    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" event={"ID":"c16cc6ed-d030-440f-97dc-49d67415754b","Type":"ContainerStarted","Data":"d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c"}
Oct 30 14:32:14.063892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:14.063658    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:15.099960 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.099756    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"]
Oct 30 14:32:15.100799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.100763    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p" podUID="d33b2b99-5155-44d9-b24f-e0de33e3e2fe" containerName="maas-api" containerID="cri-o://1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc" gracePeriod=30
Oct 30 14:32:15.456134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.456096    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:32:15.574857 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.574813    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-openshift-service-ca\") pod \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") "
Oct 30 14:32:15.575042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.574898    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-metrics-certs\") pod \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") "
Oct 30 14:32:15.575092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.575053    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-maas-api-tls\") pod \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") "
Oct 30 14:32:15.575148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.575132    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-kube-api-access-zxqdt\" (UniqueName: \"kubernetes.io/projected/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-kube-api-access-zxqdt\") pod \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\" (UID: \"d33b2b99-5155-44d9-b24f-e0de33e3e2fe\") "
Oct 30 14:32:15.575263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.575231    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-openshift-service-ca" pod "d33b2b99-5155-44d9-b24f-e0de33e3e2fe" (UID: "d33b2b99-5155-44d9-b24f-e0de33e3e2fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:32:15.575799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.575778    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:15.577080 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.577053    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-metrics-certs" pod "d33b2b99-5155-44d9-b24f-e0de33e3e2fe" (UID: "d33b2b99-5155-44d9-b24f-e0de33e3e2fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:32:15.577226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.577205    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-maas-api-tls" pod "d33b2b99-5155-44d9-b24f-e0de33e3e2fe" (UID: "d33b2b99-5155-44d9-b24f-e0de33e3e2fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:32:15.577273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.577210    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-kube-api-access-zxqdt" pod "d33b2b99-5155-44d9-b24f-e0de33e3e2fe" (UID: "d33b2b99-5155-44d9-b24f-e0de33e3e2fe"). InnerVolumeSpecName "kube-api-access-zxqdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:32:15.682924 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.682839    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:15.683132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.683116    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:15.683248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:15.683232    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxqdt\" (UniqueName: \"kubernetes.io/projected/d33b2b99-5155-44d9-b24f-e0de33e3e2fe-kube-api-access-zxqdt\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:16.078961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:16.078924    4530 generic.go:356] "Generic (PLEG): container finished" podID="d33b2b99-5155-44d9-b24f-e0de33e3e2fe" containerID="1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc" exitCode=0
Oct 30 14:32:16.079135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:16.078964    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"
Oct 30 14:32:16.079135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:16.078983    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p" event={"ID":"d33b2b99-5155-44d9-b24f-e0de33e3e2fe","Type":"ContainerDied","Data":"1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc"}
Oct 30 14:32:16.079135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:16.079026    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p" event={"ID":"d33b2b99-5155-44d9-b24f-e0de33e3e2fe","Type":"ContainerDied","Data":"e18027f11cf815a30e0374ee5def699a689c57ee89b7ab9b960d57643ca9c9c4"}
Oct 30 14:32:16.079135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:16.079048    4530 scope.go:122] "RemoveContainer" containerID="1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc"
Oct 30 14:32:16.091130 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:16.091112    4530 scope.go:122] "RemoveContainer" containerID="1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc"
Oct 30 14:32:16.091403 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:16.091382    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc\": container with ID starting with 1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc not found: ID does not exist" containerID="1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc"
Oct 30 14:32:16.091470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:16.091410    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc"} err="failed to get container status \"1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc\": rpc error: code = NotFound desc = could not find container \"1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc\": container with ID starting with 1dfad399340b8e2bec363a54000fa373479dac0d3504deefc3fe57676199f3cc not found: ID does not exist"
Oct 30 14:32:16.141874 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:16.141851    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"]
Oct 30 14:32:16.156132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:16.156111    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-34bcc5-77fc67bc7c-qh59p"]
Oct 30 14:32:17.116290 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:17.116253    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33b2b99-5155-44d9-b24f-e0de33e3e2fe" path="/var/lib/kubelet/pods/d33b2b99-5155-44d9-b24f-e0de33e3e2fe/volumes"
Oct 30 14:32:17.605647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:17.605615    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"]
Oct 30 14:32:20.072001 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:20.071961    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.249:8443/health\": dial tcp 10.133.0.249:8443: connect: connection refused"
Oct 30 14:32:24.152814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:24.152780    4530 generic.go:356] "Generic (PLEG): container finished" podID="c16cc6ed-d030-440f-97dc-49d67415754b" containerID="d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c" exitCode=1
Oct 30 14:32:24.153270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:24.152852    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" event={"ID":"c16cc6ed-d030-440f-97dc-49d67415754b","Type":"ContainerDied","Data":"d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c"}
Oct 30 14:32:24.153270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:24.152922    4530 scope.go:122] "RemoveContainer" containerID="b5c232842b69d0882f49b3fcb10ebc576eaf0f6d8739eb6cff28f70b8b02ae4d"
Oct 30 14:32:24.153270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:24.153266    4530 scope.go:122] "RemoveContainer" containerID="d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c"
Oct 30 14:32:24.153629 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:24.153601    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw_odh-ai-gateway-infra(c16cc6ed-d030-440f-97dc-49d67415754b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" podUID="c16cc6ed-d030-440f-97dc-49d67415754b"
Oct 30 14:32:24.593713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:24.593681    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"]
Oct 30 14:32:24.593953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:24.593930    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy" containerID="cri-o://4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d" gracePeriod=30
Oct 30 14:32:25.500536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.500508    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"]
Oct 30 14:32:25.501441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.501418    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d33b2b99-5155-44d9-b24f-e0de33e3e2fe" containerName="maas-api"
Oct 30 14:32:25.501536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.501444    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33b2b99-5155-44d9-b24f-e0de33e3e2fe" containerName="maas-api"
Oct 30 14:32:25.501536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.501500    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e02d6098-1f27-4b24-b0bc-49ff80d17724" containerName="main"
Oct 30 14:32:25.501536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.501509    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e02d6098-1f27-4b24-b0bc-49ff80d17724" containerName="main"
Oct 30 14:32:25.501536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.501524    4530 cpu_manager.go:414] "containerMap: removing container" podUID="a381d859-fd85-4ec4-be51-3f6d6902cd01" containerName="maas-api"
Oct 30 14:32:25.501536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.501531    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a381d859-fd85-4ec4-be51-3f6d6902cd01" containerName="maas-api"
Oct 30 14:32:25.501770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.501751    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33b2b99-5155-44d9-b24f-e0de33e3e2fe" containerName="maas-api"
Oct 30 14:32:25.501813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.501776    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e02d6098-1f27-4b24-b0bc-49ff80d17724" containerName="main"
Oct 30 14:32:25.501813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.501786    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="a381d859-fd85-4ec4-be51-3f6d6902cd01" containerName="maas-api"
Oct 30 14:32:25.501813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.501803    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdd5958-7f02-448b-8bcc-654433ec1cd2" containerName="istio-proxy"
Oct 30 14:32:25.507262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.507240    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.528171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.528145    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"]
Oct 30 14:32:25.535915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.535868    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-1d6347\"/\"auth-test-model-1d6347-kserve-self-signed-certs\""
Oct 30 14:32:25.535915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.535883    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-1d6347\"/\"default-dockercfg-6rw8l\""
Oct 30 14:32:25.538791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.538766    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-1d6347\"/\"openshift-service-ca.crt\""
Oct 30 14:32:25.538906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.538808    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-1d6347\"/\"kube-root-ca.crt\""
Oct 30 14:32:25.673003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.672973    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpgqk\" (UniqueName: \"kubernetes.io/projected/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-kube-api-access-cpgqk\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.673139 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.673089    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tls-certs\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.673184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.673169    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-model-cache\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.673222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.673201    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tmp-dir\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.673263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.673222    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-dshm\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.673302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.673261    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-home\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.774341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.774266    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpgqk\" (UniqueName: \"kubernetes.io/projected/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-kube-api-access-cpgqk\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.774500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.774375    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tls-certs\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.774500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.774472    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-model-cache\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.774665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.774514    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tmp-dir\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.774665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.774561    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-dshm\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.774665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.774610    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-home\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.775001 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.774973    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-model-cache\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.775105 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.775019    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tmp-dir\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.775105 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.775061    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-home\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.776917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.776897    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-dshm\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.777228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.777208    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tls-certs\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.814925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.814892    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpgqk\" (UniqueName: \"kubernetes.io/projected/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-kube-api-access-cpgqk\") pod \"auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") " pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:25.833869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:25.833831    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:26.229453 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:32:26.229404    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8979a6ea_2a77_43e3_8c28_157cf5b4f0f9.slice/crio-ea27ae4f50d224caa6c9f996178c6e8d7738c11855b2f519513a299fa87df79a WatchSource:0}: Error finding container ea27ae4f50d224caa6c9f996178c6e8d7738c11855b2f519513a299fa87df79a: Status 404 returned error can't find the container with id ea27ae4f50d224caa6c9f996178c6e8d7738c11855b2f519513a299fa87df79a
Oct 30 14:32:26.236291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:26.235616    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"]
Oct 30 14:32:27.181193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:27.181148    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2" event={"ID":"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9","Type":"ContainerStarted","Data":"3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf"}
Oct 30 14:32:27.181719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:27.181207    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2" event={"ID":"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9","Type":"ContainerStarted","Data":"ea27ae4f50d224caa6c9f996178c6e8d7738c11855b2f519513a299fa87df79a"}
Oct 30 14:32:27.233879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:27.233831    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2" podStartSLOduration=2.233818679 podStartE2EDuration="2.233818679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:32:27.232134223 +0000 UTC m=+2612.264033162" watchObservedRunningTime="2026-10-30 14:32:27.233818679 +0000 UTC m=+2612.265717632"
Oct 30 14:32:27.834686 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:27.834656    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:29.878674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:29.878644    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:32:30.019500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.019468    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-envoy\") pod \"3310a9c6-08cc-46ef-8a4a-27e960c28216\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") "
Oct 30 14:32:30.019693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.019591    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-credential-socket\") pod \"3310a9c6-08cc-46ef-8a4a-27e960c28216\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") "
Oct 30 14:32:30.019693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.019616    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-data\") pod \"3310a9c6-08cc-46ef-8a4a-27e960c28216\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") "
Oct 30 14:32:30.019693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.019641    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-token\" (UniqueName: \"kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-token\") pod \"3310a9c6-08cc-46ef-8a4a-27e960c28216\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") "
Oct 30 14:32:30.019693 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.019665    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-certs\") pod \"3310a9c6-08cc-46ef-8a4a-27e960c28216\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") "
Oct 30 14:32:30.019906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.019752    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-socket\") pod \"3310a9c6-08cc-46ef-8a4a-27e960c28216\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") "
Oct 30 14:32:30.019906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.019828    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-podinfo\") pod \"3310a9c6-08cc-46ef-8a4a-27e960c28216\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") "
Oct 30 14:32:30.019906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.019881    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3310a9c6-08cc-46ef-8a4a-27e960c28216-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3310a9c6-08cc-46ef-8a4a-27e960c28216-istiod-ca-cert\") pod \"3310a9c6-08cc-46ef-8a4a-27e960c28216\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") "
Oct 30 14:32:30.020074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.019913    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-credential-socket" pod "3310a9c6-08cc-46ef-8a4a-27e960c28216" (UID: "3310a9c6-08cc-46ef-8a4a-27e960c28216"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:30.020074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.019933    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-data" pod "3310a9c6-08cc-46ef-8a4a-27e960c28216" (UID: "3310a9c6-08cc-46ef-8a4a-27e960c28216"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:30.020074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.019958    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-kube-api-access-xsc5q\" (UniqueName: \"kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-kube-api-access-xsc5q\") pod \"3310a9c6-08cc-46ef-8a4a-27e960c28216\" (UID: \"3310a9c6-08cc-46ef-8a4a-27e960c28216\") "
Oct 30 14:32:30.020074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.020045    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-socket" pod "3310a9c6-08cc-46ef-8a4a-27e960c28216" (UID: "3310a9c6-08cc-46ef-8a4a-27e960c28216"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:30.020273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.020084    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-certs" pod "3310a9c6-08cc-46ef-8a4a-27e960c28216" (UID: "3310a9c6-08cc-46ef-8a4a-27e960c28216"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:30.020333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.020297    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3310a9c6-08cc-46ef-8a4a-27e960c28216-istiod-ca-cert" pod "3310a9c6-08cc-46ef-8a4a-27e960c28216" (UID: "3310a9c6-08cc-46ef-8a4a-27e960c28216"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:32:30.020529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.020512    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:30.020529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.020529    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:30.020720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.020568    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:30.020720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.020581    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:30.020720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.020591    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3310a9c6-08cc-46ef-8a4a-27e960c28216-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:30.022167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.022137    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-token" pod "3310a9c6-08cc-46ef-8a4a-27e960c28216" (UID: "3310a9c6-08cc-46ef-8a4a-27e960c28216"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:32:30.022285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.022175    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-kube-api-access-xsc5q" pod "3310a9c6-08cc-46ef-8a4a-27e960c28216" (UID: "3310a9c6-08cc-46ef-8a4a-27e960c28216"). InnerVolumeSpecName "kube-api-access-xsc5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:32:30.022285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.022254    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-envoy" pod "3310a9c6-08cc-46ef-8a4a-27e960c28216" (UID: "3310a9c6-08cc-46ef-8a4a-27e960c28216"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:30.022409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.022330    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-podinfo" pod "3310a9c6-08cc-46ef-8a4a-27e960c28216" (UID: "3310a9c6-08cc-46ef-8a4a-27e960c28216"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:32:30.121469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.121406    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:30.121469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.121432    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:30.121469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.121441    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3310a9c6-08cc-46ef-8a4a-27e960c28216-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:30.121469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.121452    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsc5q\" (UniqueName: \"kubernetes.io/projected/3310a9c6-08cc-46ef-8a4a-27e960c28216-kube-api-access-xsc5q\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:30.205434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.205402    4530 generic.go:356] "Generic (PLEG): container finished" podID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerID="4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d" exitCode=0
Oct 30 14:32:30.205594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.205450    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"
Oct 30 14:32:30.205594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.205485    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" event={"ID":"3310a9c6-08cc-46ef-8a4a-27e960c28216","Type":"ContainerDied","Data":"4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d"}
Oct 30 14:32:30.205594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.205555    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr" event={"ID":"3310a9c6-08cc-46ef-8a4a-27e960c28216","Type":"ContainerDied","Data":"0c74ce2d8df920abe6f2f23f62b39b51ff1bd761512ae1fa49eba7ee29b3edb1"}
Oct 30 14:32:30.205594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.205568    4530 scope.go:122] "RemoveContainer" containerID="4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d"
Oct 30 14:32:30.217036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.217021    4530 scope.go:122] "RemoveContainer" containerID="601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3"
Oct 30 14:32:30.226849 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.226834    4530 scope.go:122] "RemoveContainer" containerID="4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d"
Oct 30 14:32:30.227118 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:30.227099    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d\": container with ID starting with 4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d not found: ID does not exist" containerID="4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d"
Oct 30 14:32:30.227192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.227129    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d"} err="failed to get container status \"4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d\": rpc error: code = NotFound desc = could not find container \"4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d\": container with ID starting with 4c638fbd6e0fd90a04af1a0307fa2a54ae19324c511a132d5e4e8b4ff09a885d not found: ID does not exist"
Oct 30 14:32:30.227192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.227152    4530 scope.go:122] "RemoveContainer" containerID="601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3"
Oct 30 14:32:30.227396 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:30.227375    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3\": container with ID starting with 601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3 not found: ID does not exist" containerID="601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3"
Oct 30 14:32:30.227445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.227402    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3"} err="failed to get container status \"601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3\": rpc error: code = NotFound desc = could not find container \"601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3\": container with ID starting with 601edc07ab48dcec3333a6d5e5102ac674e77db85421681481dc7a176332a4d3 not found: ID does not exist"
Oct 30 14:32:30.269342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.269320    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"]
Oct 30 14:32:30.273356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:30.273333    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-34bcc5-openshift-default-74b75878bc-ddnhr"]
Oct 30 14:32:31.115687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:31.115659    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" path="/var/lib/kubelet/pods/3310a9c6-08cc-46ef-8a4a-27e960c28216/volumes"
Oct 30 14:32:31.921507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:31.921466    4530 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:31.921986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:31.921963    4530 scope.go:122] "RemoveContainer" containerID="d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c"
Oct 30 14:32:31.922306 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:31.922284    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw_odh-ai-gateway-infra(c16cc6ed-d030-440f-97dc-49d67415754b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" podUID="c16cc6ed-d030-440f-97dc-49d67415754b"
Oct 30 14:32:32.627513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:32.627449    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k" podUID="4c33c19b-0057-40dd-af41-fe69990fb374" containerName="main" containerID="cri-o://faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f" gracePeriod=45
Oct 30 14:32:32.910731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:32.910701    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:32:33.055811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.055779    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c33c19b-0057-40dd-af41-fe69990fb374-kube-api-access-k9wbw\" (UniqueName: \"kubernetes.io/projected/4c33c19b-0057-40dd-af41-fe69990fb374-kube-api-access-k9wbw\") pod \"4c33c19b-0057-40dd-af41-fe69990fb374\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") "
Oct 30 14:32:33.056000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.055850    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-model-cache\") pod \"4c33c19b-0057-40dd-af41-fe69990fb374\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") "
Oct 30 14:32:33.056000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.055920    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c33c19b-0057-40dd-af41-fe69990fb374-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c33c19b-0057-40dd-af41-fe69990fb374-tls-certs\") pod \"4c33c19b-0057-40dd-af41-fe69990fb374\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") "
Oct 30 14:32:33.056000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.055991    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-tmp-dir\") pod \"4c33c19b-0057-40dd-af41-fe69990fb374\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") "
Oct 30 14:32:33.056173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.056121    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-dshm\") pod \"4c33c19b-0057-40dd-af41-fe69990fb374\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") "
Oct 30 14:32:33.056173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.056141    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-model-cache" pod "4c33c19b-0057-40dd-af41-fe69990fb374" (UID: "4c33c19b-0057-40dd-af41-fe69990fb374"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:33.056307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.056250    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-tmp-dir" pod "4c33c19b-0057-40dd-af41-fe69990fb374" (UID: "4c33c19b-0057-40dd-af41-fe69990fb374"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:33.056307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.056269    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-home\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-home\") pod \"4c33c19b-0057-40dd-af41-fe69990fb374\" (UID: \"4c33c19b-0057-40dd-af41-fe69990fb374\") "
Oct 30 14:32:33.056497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.056470    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-home" pod "4c33c19b-0057-40dd-af41-fe69990fb374" (UID: "4c33c19b-0057-40dd-af41-fe69990fb374"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:33.056892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.056857    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:33.057016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.056895    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:33.057016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.056910    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:33.058187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.058158    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c33c19b-0057-40dd-af41-fe69990fb374-kube-api-access-k9wbw" pod "4c33c19b-0057-40dd-af41-fe69990fb374" (UID: "4c33c19b-0057-40dd-af41-fe69990fb374"). InnerVolumeSpecName "kube-api-access-k9wbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:32:33.058323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.058187    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-dshm" pod "4c33c19b-0057-40dd-af41-fe69990fb374" (UID: "4c33c19b-0057-40dd-af41-fe69990fb374"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:33.058323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.058196    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c33c19b-0057-40dd-af41-fe69990fb374-tls-certs" pod "4c33c19b-0057-40dd-af41-fe69990fb374" (UID: "4c33c19b-0057-40dd-af41-fe69990fb374"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:32:33.158823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.158759    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9wbw\" (UniqueName: \"kubernetes.io/projected/4c33c19b-0057-40dd-af41-fe69990fb374-kube-api-access-k9wbw\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:33.158823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.158791    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c33c19b-0057-40dd-af41-fe69990fb374-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:33.158823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.158803    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c33c19b-0057-40dd-af41-fe69990fb374-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:33.232293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.232266    4530 generic.go:356] "Generic (PLEG): container finished" podID="4c33c19b-0057-40dd-af41-fe69990fb374" containerID="faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f" exitCode=0
Oct 30 14:32:33.232417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.232306    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"
Oct 30 14:32:33.232417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.232351    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k" event={"ID":"4c33c19b-0057-40dd-af41-fe69990fb374","Type":"ContainerDied","Data":"faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f"}
Oct 30 14:32:33.232417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.232397    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k" event={"ID":"4c33c19b-0057-40dd-af41-fe69990fb374","Type":"ContainerDied","Data":"251ae85fa071b9bbe24c004be3bf747cc6d357996a68d822cae369767adc5ea4"}
Oct 30 14:32:33.232649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.232418    4530 scope.go:122] "RemoveContainer" containerID="faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f"
Oct 30 14:32:33.242700 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.242674    4530 scope.go:122] "RemoveContainer" containerID="faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f"
Oct 30 14:32:33.242971 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:33.242952    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f\": container with ID starting with faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f not found: ID does not exist" containerID="faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f"
Oct 30 14:32:33.243029 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.242982    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f"} err="failed to get container status \"faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f\": rpc error: code = NotFound desc = could not find container \"faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f\": container with ID starting with faa798448a05500516f689a392e687819768f4263b0731c95220c133a397542f not found: ID does not exist"
Oct 30 14:32:33.295164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.295140    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"]
Oct 30 14:32:33.302548 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.302514    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0b1c15/test-model-0b1c15-kserve-7756859875-snm9k"]
Oct 30 14:32:33.437874 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.437775    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"]
Oct 30 14:32:33.438083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:33.438037    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerName="istio-proxy" containerID="cri-o://7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8" gracePeriod=30
Oct 30 14:32:34.108057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:34.108026    4530 scope.go:122] "RemoveContainer" containerID="d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c"
Oct 30 14:32:35.116187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:35.116156    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c33c19b-0057-40dd-af41-fe69990fb374" path="/var/lib/kubelet/pods/4c33c19b-0057-40dd-af41-fe69990fb374/volumes"
Oct 30 14:32:35.249692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:35.249659    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" event={"ID":"c16cc6ed-d030-440f-97dc-49d67415754b","Type":"ContainerStarted","Data":"dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee"}
Oct 30 14:32:35.249882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:35.249864    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:35.834608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:35.834577    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:35.847392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:35.847362    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:36.268256 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:36.268174    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:32:38.712888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.712824    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:32:38.804625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.804593    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-kube-api-access-88zwb\" (UniqueName: \"kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-kube-api-access-88zwb\") pod \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") "
Oct 30 14:32:38.804625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.804630    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-socket\") pod \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") "
Oct 30 14:32:38.804876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.804680    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-data\") pod \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") "
Oct 30 14:32:38.804876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.804781    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-podinfo\") pod \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") "
Oct 30 14:32:38.804876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.804835    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istiod-ca-cert\") pod \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") "
Oct 30 14:32:38.805063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.804896    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-envoy\") pod \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") "
Oct 30 14:32:38.805063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.804977    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-token\" (UniqueName: \"kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-token\") pod \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") "
Oct 30 14:32:38.805063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.805028    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-credential-socket\") pod \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") "
Oct 30 14:32:38.805232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.805071    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-certs\") pod \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\" (UID: \"467f07a9-7aa2-4f27-9d2e-11106f3deb44\") "
Oct 30 14:32:38.805232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.805186    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-data" pod "467f07a9-7aa2-4f27-9d2e-11106f3deb44" (UID: "467f07a9-7aa2-4f27-9d2e-11106f3deb44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:38.805381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.805361    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-socket" pod "467f07a9-7aa2-4f27-9d2e-11106f3deb44" (UID: "467f07a9-7aa2-4f27-9d2e-11106f3deb44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:38.805697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.805670    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:38.805697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.805698    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:38.806418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.806379    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-credential-socket" pod "467f07a9-7aa2-4f27-9d2e-11106f3deb44" (UID: "467f07a9-7aa2-4f27-9d2e-11106f3deb44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:38.806568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.806449    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istiod-ca-cert" pod "467f07a9-7aa2-4f27-9d2e-11106f3deb44" (UID: "467f07a9-7aa2-4f27-9d2e-11106f3deb44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:32:38.806675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.806653    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-certs" pod "467f07a9-7aa2-4f27-9d2e-11106f3deb44" (UID: "467f07a9-7aa2-4f27-9d2e-11106f3deb44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:38.808106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.808082    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-envoy" pod "467f07a9-7aa2-4f27-9d2e-11106f3deb44" (UID: "467f07a9-7aa2-4f27-9d2e-11106f3deb44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:38.808264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.808232    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-kube-api-access-88zwb" pod "467f07a9-7aa2-4f27-9d2e-11106f3deb44" (UID: "467f07a9-7aa2-4f27-9d2e-11106f3deb44"). InnerVolumeSpecName "kube-api-access-88zwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:32:38.809509 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.809487    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-podinfo" pod "467f07a9-7aa2-4f27-9d2e-11106f3deb44" (UID: "467f07a9-7aa2-4f27-9d2e-11106f3deb44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:32:38.809649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.809631    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-token" pod "467f07a9-7aa2-4f27-9d2e-11106f3deb44" (UID: "467f07a9-7aa2-4f27-9d2e-11106f3deb44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:32:38.907292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.907266    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:38.907292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.907289    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:38.907448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.907298    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:38.907448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.907307    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:38.907448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.907316    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:38.907448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.907324    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467f07a9-7aa2-4f27-9d2e-11106f3deb44-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:38.907448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:38.907332    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88zwb\" (UniqueName: \"kubernetes.io/projected/467f07a9-7aa2-4f27-9d2e-11106f3deb44-kube-api-access-88zwb\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:39.280182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:39.280150    4530 generic.go:356] "Generic (PLEG): container finished" podID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerID="7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8" exitCode=0
Oct 30 14:32:39.280337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:39.280201    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"
Oct 30 14:32:39.280337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:39.280237    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" event={"ID":"467f07a9-7aa2-4f27-9d2e-11106f3deb44","Type":"ContainerDied","Data":"7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8"}
Oct 30 14:32:39.280337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:39.280292    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp" event={"ID":"467f07a9-7aa2-4f27-9d2e-11106f3deb44","Type":"ContainerDied","Data":"ea7cb4fb26e35d9d9160b67f72b3596cbb9be9a2262c4614d385013d8ba3e3c9"}
Oct 30 14:32:39.280337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:39.280321    4530 scope.go:122] "RemoveContainer" containerID="7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8"
Oct 30 14:32:39.291099 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:39.291080    4530 scope.go:122] "RemoveContainer" containerID="487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312"
Oct 30 14:32:39.300688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:39.300669    4530 scope.go:122] "RemoveContainer" containerID="7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8"
Oct 30 14:32:39.300984 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:39.300955    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8\": container with ID starting with 7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8 not found: ID does not exist" containerID="7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8"
Oct 30 14:32:39.301060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:39.300995    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8"} err="failed to get container status \"7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8\": rpc error: code = NotFound desc = could not find container \"7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8\": container with ID starting with 7429f01efdc884ade26d028063793a594426128d68a72994d3e48605a76b87c8 not found: ID does not exist"
Oct 30 14:32:39.301060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:39.301019    4530 scope.go:122] "RemoveContainer" containerID="487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312"
Oct 30 14:32:39.301278 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:39.301261    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312\": container with ID starting with 487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312 not found: ID does not exist" containerID="487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312"
Oct 30 14:32:39.301318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:39.301291    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312"} err="failed to get container status \"487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312\": rpc error: code = NotFound desc = could not find container \"487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312\": container with ID starting with 487518570dbd149beef5f1d33371c8bb2ebae088ac46346f31cfaf3907351312 not found: ID does not exist"
Oct 30 14:32:39.364851 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:39.364776    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"]
Oct 30 14:32:39.369173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:39.369143    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-409364-openshift-default-868c776847-ntjrp"]
Oct 30 14:32:41.115862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:41.115828    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" path="/var/lib/kubelet/pods/467f07a9-7aa2-4f27-9d2e-11106f3deb44/volumes"
Oct 30 14:32:41.262824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:41.262793    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:32:41.371885 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:41.371781    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:32:44.616820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:44.616788    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"]
Oct 30 14:32:44.617223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:44.617049    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs" podUID="c7e10d2d-7b12-4cad-bcb9-b9582142b2b8" containerName="maas-api" containerID="cri-o://746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91" gracePeriod=30
Oct 30 14:32:44.998400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:44.998275    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:32:45.105687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.105641    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-kube-api-access-vwgm2\" (UniqueName: \"kubernetes.io/projected/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-kube-api-access-vwgm2\") pod \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") "
Oct 30 14:32:45.105687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.105691    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-metrics-certs\") pod \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") "
Oct 30 14:32:45.105963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.105766    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-openshift-service-ca\") pod \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") "
Oct 30 14:32:45.105963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.105843    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-maas-api-tls\") pod \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\" (UID: \"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8\") "
Oct 30 14:32:45.106219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.106191    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-openshift-service-ca" pod "c7e10d2d-7b12-4cad-bcb9-b9582142b2b8" (UID: "c7e10d2d-7b12-4cad-bcb9-b9582142b2b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:32:45.106511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.106484    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:45.108344 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.108313    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-maas-api-tls" pod "c7e10d2d-7b12-4cad-bcb9-b9582142b2b8" (UID: "c7e10d2d-7b12-4cad-bcb9-b9582142b2b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:32:45.108444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.108392    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-metrics-certs" pod "c7e10d2d-7b12-4cad-bcb9-b9582142b2b8" (UID: "c7e10d2d-7b12-4cad-bcb9-b9582142b2b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:32:45.108513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.108446    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-kube-api-access-vwgm2" pod "c7e10d2d-7b12-4cad-bcb9-b9582142b2b8" (UID: "c7e10d2d-7b12-4cad-bcb9-b9582142b2b8"). InnerVolumeSpecName "kube-api-access-vwgm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:32:45.221659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.209091    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwgm2\" (UniqueName: \"kubernetes.io/projected/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-kube-api-access-vwgm2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:45.221659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.209334    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:45.221659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.209352    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:45.326556 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.326501    4530 generic.go:356] "Generic (PLEG): container finished" podID="c7e10d2d-7b12-4cad-bcb9-b9582142b2b8" containerID="746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91" exitCode=0
Oct 30 14:32:45.326738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.326610    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"
Oct 30 14:32:45.326738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.326613    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs" event={"ID":"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8","Type":"ContainerDied","Data":"746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91"}
Oct 30 14:32:45.326738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.326684    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs" event={"ID":"c7e10d2d-7b12-4cad-bcb9-b9582142b2b8","Type":"ContainerDied","Data":"f04deb4522d768fa9b4ba06254994219c1a3e2c2a3f6ae8bf811f74198356c84"}
Oct 30 14:32:45.326738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.326716    4530 scope.go:122] "RemoveContainer" containerID="746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91"
Oct 30 14:32:45.337025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.337008    4530 scope.go:122] "RemoveContainer" containerID="746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91"
Oct 30 14:32:45.337340 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:32:45.337318    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91\": container with ID starting with 746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91 not found: ID does not exist" containerID="746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91"
Oct 30 14:32:45.337391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.337350    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91"} err="failed to get container status \"746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91\": rpc error: code = NotFound desc = could not find container \"746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91\": container with ID starting with 746a240ce2d9053813cd77b28dc0c38ac3b29841fdc6ef385725cbc8115b0f91 not found: ID does not exist"
Oct 30 14:32:45.416672 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.416610    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"]
Oct 30 14:32:45.429395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:45.429342    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f3c580-5f847ddc7d-tczcs"]
Oct 30 14:32:47.167620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:47.164680    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7e10d2d-7b12-4cad-bcb9-b9582142b2b8" path="/var/lib/kubelet/pods/c7e10d2d-7b12-4cad-bcb9-b9582142b2b8/volumes"
Oct 30 14:32:51.806616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.806583    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"]
Oct 30 14:32:51.807434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807412    4530 cpu_manager.go:414] "containerMap: removing container" podUID="4c33c19b-0057-40dd-af41-fe69990fb374" containerName="main"
Oct 30 14:32:51.807564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807436    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c33c19b-0057-40dd-af41-fe69990fb374" containerName="main"
Oct 30 14:32:51.807564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807454    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c7e10d2d-7b12-4cad-bcb9-b9582142b2b8" containerName="maas-api"
Oct 30 14:32:51.807564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807462    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e10d2d-7b12-4cad-bcb9-b9582142b2b8" containerName="maas-api"
Oct 30 14:32:51.807564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807486    4530 cpu_manager.go:414] "containerMap: removing container" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerName="istio-proxy"
Oct 30 14:32:51.807564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807494    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerName="istio-proxy"
Oct 30 14:32:51.807564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807507    4530 cpu_manager.go:414] "containerMap: removing container" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy"
Oct 30 14:32:51.807564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807514    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy"
Oct 30 14:32:51.807564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807526    4530 cpu_manager.go:414] "containerMap: removing container" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerName="istio-proxy"
Oct 30 14:32:51.807564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807533    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerName="istio-proxy"
Oct 30 14:32:51.808056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807759    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy"
Oct 30 14:32:51.808056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807777    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c33c19b-0057-40dd-af41-fe69990fb374" containerName="main"
Oct 30 14:32:51.808056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807787    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e10d2d-7b12-4cad-bcb9-b9582142b2b8" containerName="maas-api"
Oct 30 14:32:51.808056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807801    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerName="istio-proxy"
Oct 30 14:32:51.808056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807812    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy"
Oct 30 14:32:51.808056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.807822    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="467f07a9-7aa2-4f27-9d2e-11106f3deb44" containerName="istio-proxy"
Oct 30 14:32:51.813261 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.813237    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:51.832324 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.832297    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"]
Oct 30 14:32:51.842275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.842250    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a59ecb\"/\"default-dockercfg-6mpvj\""
Oct 30 14:32:51.842417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.842365    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a59ecb\"/\"auth-test-model-a59ecb-kserve-self-signed-certs\""
Oct 30 14:32:51.844823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.844801    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a59ecb\"/\"kube-root-ca.crt\""
Oct 30 14:32:51.844823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.844837    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a59ecb\"/\"openshift-service-ca.crt\""
Oct 30 14:32:51.922422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.922393    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tmp-dir\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:51.922597 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.922433    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-home\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:51.922688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.922666    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tls-certs\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:51.922817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.922794    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b2cr\" (UniqueName: \"kubernetes.io/projected/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-kube-api-access-6b2cr\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:51.922949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.922927    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-model-cache\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:51.923011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:51.922974    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-dshm\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.055564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.041598    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tls-certs\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.055564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.041712    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b2cr\" (UniqueName: \"kubernetes.io/projected/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-kube-api-access-6b2cr\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.055564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.041803    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-model-cache\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.055564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.041839    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-dshm\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.055564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.041915    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tmp-dir\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.055564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.041950    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-home\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.055564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.042441    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-home\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.055564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.043173    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-model-cache\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.055564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.048165    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-dshm\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.055564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.050988    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tmp-dir\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.068610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.064290    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tls-certs\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.088560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.079289    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b2cr\" (UniqueName: \"kubernetes.io/projected/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-kube-api-access-6b2cr\") pod \"auth-test-model-a59ecb-kserve-85469cf445-rvqpx\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.143074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.138202    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:52.458220 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:32:52.458183    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f78d99a_71a0_4ed7_88fb_7fc54fada30d.slice/crio-74ec0187af3f5f3f230bb6ed1308668209f474a77bfe8a3e9e926717eb7b3a5d WatchSource:0}: Error finding container 74ec0187af3f5f3f230bb6ed1308668209f474a77bfe8a3e9e926717eb7b3a5d: Status 404 returned error can't find the container with id 74ec0187af3f5f3f230bb6ed1308668209f474a77bfe8a3e9e926717eb7b3a5d
Oct 30 14:32:52.458980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.458931    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"]
Oct 30 14:32:52.554986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.554949    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"]
Oct 30 14:32:52.555961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.555870    4530 cpu_manager.go:414] "containerMap: removing container" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy"
Oct 30 14:32:52.555961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.555892    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3310a9c6-08cc-46ef-8a4a-27e960c28216" containerName="istio-proxy"
Oct 30 14:32:52.559989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.559958    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.580325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.580302    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"]
Oct 30 14:32:52.588250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.588213    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6e91df-openshift-default-dockercfg-plsv2\""
Oct 30 14:32:52.760619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.752998    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffcmt\" (UniqueName: \"kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-kube-api-access-ffcmt\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.760619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.753125    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istiod-ca-cert\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.760619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.753274    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-envoy\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.760619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.753437    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-data\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.760619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.753550    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-certs\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.760619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.753630    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-socket\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.760619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.753693    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-token\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.760619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.753730    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-credential-socket\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.760619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.753809    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-podinfo\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.860565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.860392    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-certs\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.860565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.860480    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-socket\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.860565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.860557    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-token\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.861194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.860592    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-credential-socket\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.861194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.860662    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-podinfo\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.861194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.860717    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffcmt\" (UniqueName: \"kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-kube-api-access-ffcmt\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.861194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.860777    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istiod-ca-cert\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.861194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.860813    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-envoy\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.861194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.860842    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-data\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.861565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.861351    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-data\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.861637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.861607    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-certs\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.869619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.861822    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-socket\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.869619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.862332    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-credential-socket\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.869619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.865571    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istiod-ca-cert\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.872627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.870154    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-podinfo\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.872627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.871574    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-envoy\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.903742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.902718    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffcmt\" (UniqueName: \"kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-kube-api-access-ffcmt\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:52.903742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:52.903667    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-token\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:53.019212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.019119    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"]
Oct 30 14:32:53.019415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.019384    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw" podUID="c913bf32-ca38-48b7-98ec-b4ed589ab093" containerName="maas-api" containerID="cri-o://e65516694a12afda3270e99d269680117f1285ee46035fab3a98df7c7404b7a5" gracePeriod=30
Oct 30 14:32:53.179423 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.179389    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:53.420565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.415394    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx" event={"ID":"6f78d99a-71a0-4ed7-88fb-7fc54fada30d","Type":"ContainerStarted","Data":"58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff"}
Oct 30 14:32:53.421278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.421037    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx" event={"ID":"6f78d99a-71a0-4ed7-88fb-7fc54fada30d","Type":"ContainerStarted","Data":"74ec0187af3f5f3f230bb6ed1308668209f474a77bfe8a3e9e926717eb7b3a5d"}
Oct 30 14:32:53.423785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.423578    4530 generic.go:356] "Generic (PLEG): container finished" podID="c913bf32-ca38-48b7-98ec-b4ed589ab093" containerID="e65516694a12afda3270e99d269680117f1285ee46035fab3a98df7c7404b7a5" exitCode=0
Oct 30 14:32:53.423785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.423714    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw" event={"ID":"c913bf32-ca38-48b7-98ec-b4ed589ab093","Type":"ContainerDied","Data":"e65516694a12afda3270e99d269680117f1285ee46035fab3a98df7c7404b7a5"}
Oct 30 14:32:53.526300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.526217    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx" podStartSLOduration=2.526195446 podStartE2EDuration="2.526195446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:32:53.507925761 +0000 UTC m=+2638.539824701" watchObservedRunningTime="2026-10-30 14:32:53.526195446 +0000 UTC m=+2638.558094387"
Oct 30 14:32:53.532670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.531791    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:32:53.636721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.616182    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c913bf32-ca38-48b7-98ec-b4ed589ab093-kube-api-access-kq8fc\" (UniqueName: \"kubernetes.io/projected/c913bf32-ca38-48b7-98ec-b4ed589ab093-kube-api-access-kq8fc\") pod \"c913bf32-ca38-48b7-98ec-b4ed589ab093\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") "
Oct 30 14:32:53.636721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.616308    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-metrics-certs\") pod \"c913bf32-ca38-48b7-98ec-b4ed589ab093\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") "
Oct 30 14:32:53.636721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.616353    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c913bf32-ca38-48b7-98ec-b4ed589ab093-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c913bf32-ca38-48b7-98ec-b4ed589ab093-openshift-service-ca\") pod \"c913bf32-ca38-48b7-98ec-b4ed589ab093\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") "
Oct 30 14:32:53.636721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.616398    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-maas-api-tls\") pod \"c913bf32-ca38-48b7-98ec-b4ed589ab093\" (UID: \"c913bf32-ca38-48b7-98ec-b4ed589ab093\") "
Oct 30 14:32:53.637915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.637533    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c913bf32-ca38-48b7-98ec-b4ed589ab093-openshift-service-ca" pod "c913bf32-ca38-48b7-98ec-b4ed589ab093" (UID: "c913bf32-ca38-48b7-98ec-b4ed589ab093"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:32:53.639215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.639186    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c913bf32-ca38-48b7-98ec-b4ed589ab093-kube-api-access-kq8fc" pod "c913bf32-ca38-48b7-98ec-b4ed589ab093" (UID: "c913bf32-ca38-48b7-98ec-b4ed589ab093"). InnerVolumeSpecName "kube-api-access-kq8fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:32:53.640739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.640720    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-metrics-certs" pod "c913bf32-ca38-48b7-98ec-b4ed589ab093" (UID: "c913bf32-ca38-48b7-98ec-b4ed589ab093"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:32:53.641732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.641706    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-maas-api-tls" pod "c913bf32-ca38-48b7-98ec-b4ed589ab093" (UID: "c913bf32-ca38-48b7-98ec-b4ed589ab093"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:32:53.736644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.736602    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq8fc\" (UniqueName: \"kubernetes.io/projected/c913bf32-ca38-48b7-98ec-b4ed589ab093-kube-api-access-kq8fc\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:53.736644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.736641    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:53.736644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.736651    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c913bf32-ca38-48b7-98ec-b4ed589ab093-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:53.736644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.736659    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c913bf32-ca38-48b7-98ec-b4ed589ab093-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:53.795840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.795804    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"]
Oct 30 14:32:53.847227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.847180    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:32:53.847495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.847468    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:32:53.847855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:53.847820    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:32:54.138807 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:54.138780    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:32:54.277323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:54.277238    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"]
Oct 30 14:32:54.277765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:54.277506    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy" containerID="cri-o://ee7fc65c1cc0cede66d222bcd7d1f5418082bd89ad06834169913ce4e14b0c35" gracePeriod=30
Oct 30 14:32:54.478652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:54.478612    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" event={"ID":"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3","Type":"ContainerStarted","Data":"6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4"}
Oct 30 14:32:54.478938 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:54.478917    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" event={"ID":"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3","Type":"ContainerStarted","Data":"85ef7fe3709211b0e89beca031b54c504d28c526c5b4d6ae604ecb78addc2224"}
Oct 30 14:32:54.483818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:54.483625    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw" event={"ID":"c913bf32-ca38-48b7-98ec-b4ed589ab093","Type":"ContainerDied","Data":"79374ffada244f9ecc20b964c850962ebf781f65c693f9508031c27de3069c51"}
Oct 30 14:32:54.483818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:54.483690    4530 scope.go:122] "RemoveContainer" containerID="e65516694a12afda3270e99d269680117f1285ee46035fab3a98df7c7404b7a5"
Oct 30 14:32:54.484149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:54.484130    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"
Oct 30 14:32:54.540992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:54.540896    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" podStartSLOduration=2.5408736530000002 podStartE2EDuration="2.540873653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:32:54.537073657 +0000 UTC m=+2639.568972599" watchObservedRunningTime="2026-10-30 14:32:54.540873653 +0000 UTC m=+2639.572772653"
Oct 30 14:32:54.582359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:54.582328    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"]
Oct 30 14:32:54.588047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:54.588024    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fde75d-8695c7f585-q76fw"]
Oct 30 14:32:55.116022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:55.115990    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c913bf32-ca38-48b7-98ec-b4ed589ab093" path="/var/lib/kubelet/pods/c913bf32-ca38-48b7-98ec-b4ed589ab093/volumes"
Oct 30 14:32:55.179520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:55.179495    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:55.185673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:55.185651    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:55.491573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:55.491463    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:55.492319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:55.492295    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:32:58.822189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:58.822152    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"]
Oct 30 14:32:58.823126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:58.823108    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c913bf32-ca38-48b7-98ec-b4ed589ab093" containerName="maas-api"
Oct 30 14:32:58.823182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:58.823129    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c913bf32-ca38-48b7-98ec-b4ed589ab093" containerName="maas-api"
Oct 30 14:32:58.823276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:58.823265    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c913bf32-ca38-48b7-98ec-b4ed589ab093" containerName="maas-api"
Oct 30 14:32:58.828970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:58.828950    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"
Oct 30 14:32:58.843665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:58.843637    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"]
Oct 30 14:32:58.849183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:58.849156    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6e91df\""
Oct 30 14:32:58.851924 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:58.851904    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6e91df-dockercfg-r7ms5\""
Oct 30 14:32:58.964565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:58.963689    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"]
Oct 30 14:32:58.984593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:58.978750    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb9vt\" (UniqueName: \"kubernetes.io/projected/52ec6eeb-78f9-4100-8390-c9688a8f94dc-kube-api-access-zb9vt\") pod \"payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g\" (UID: \"52ec6eeb-78f9-4100-8390-c9688a8f94dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"
Oct 30 14:32:58.984593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:58.978947    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ec6eeb-78f9-4100-8390-c9688a8f94dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g\" (UID: \"52ec6eeb-78f9-4100-8390-c9688a8f94dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"
Oct 30 14:32:59.003562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.001749    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"]
Oct 30 14:32:59.003562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.001870    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"
Oct 30 14:32:59.080671 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.080582    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c\" (UID: \"8240e208-5f61-45a4-b0e2-e6298ce7ab8e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"
Oct 30 14:32:59.080825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.080706    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ec6eeb-78f9-4100-8390-c9688a8f94dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g\" (UID: \"52ec6eeb-78f9-4100-8390-c9688a8f94dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"
Oct 30 14:32:59.080916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.080900    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb9vt\" (UniqueName: \"kubernetes.io/projected/52ec6eeb-78f9-4100-8390-c9688a8f94dc-kube-api-access-zb9vt\") pod \"payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g\" (UID: \"52ec6eeb-78f9-4100-8390-c9688a8f94dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"
Oct 30 14:32:59.081088 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.081056    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdrj4\" (UniqueName: \"kubernetes.io/projected/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-kube-api-access-xdrj4\") pod \"payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c\" (UID: \"8240e208-5f61-45a4-b0e2-e6298ce7ab8e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"
Oct 30 14:32:59.081570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.081505    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ec6eeb-78f9-4100-8390-c9688a8f94dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g\" (UID: \"52ec6eeb-78f9-4100-8390-c9688a8f94dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"
Oct 30 14:32:59.107314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.107280    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb9vt\" (UniqueName: \"kubernetes.io/projected/52ec6eeb-78f9-4100-8390-c9688a8f94dc-kube-api-access-zb9vt\") pod \"payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g\" (UID: \"52ec6eeb-78f9-4100-8390-c9688a8f94dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"
Oct 30 14:32:59.146728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.146695    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"
Oct 30 14:32:59.182424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.182396    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdrj4\" (UniqueName: \"kubernetes.io/projected/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-kube-api-access-xdrj4\") pod \"payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c\" (UID: \"8240e208-5f61-45a4-b0e2-e6298ce7ab8e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"
Oct 30 14:32:59.182580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.182431    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c\" (UID: \"8240e208-5f61-45a4-b0e2-e6298ce7ab8e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"
Oct 30 14:32:59.183108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.183087    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c\" (UID: \"8240e208-5f61-45a4-b0e2-e6298ce7ab8e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"
Oct 30 14:32:59.210358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.210327    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdrj4\" (UniqueName: \"kubernetes.io/projected/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-kube-api-access-xdrj4\") pod \"payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c\" (UID: \"8240e208-5f61-45a4-b0e2-e6298ce7ab8e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"
Oct 30 14:32:59.326610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.326584    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"
Oct 30 14:32:59.468763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.468732    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"]
Oct 30 14:32:59.524723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.524648    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g" event={"ID":"52ec6eeb-78f9-4100-8390-c9688a8f94dc","Type":"ContainerStarted","Data":"b81a7e0147f0e9b3acd5332697b8a58607b7deb6d479d51060ddd207514cbc09"}
Oct 30 14:32:59.527896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.527287    4530 generic.go:356] "Generic (PLEG): container finished" podID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerID="ee7fc65c1cc0cede66d222bcd7d1f5418082bd89ad06834169913ce4e14b0c35" exitCode=0
Oct 30 14:32:59.527896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.527555    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" event={"ID":"a83aae0b-4976-4a91-937c-c1e3dda7f151","Type":"ContainerDied","Data":"ee7fc65c1cc0cede66d222bcd7d1f5418082bd89ad06834169913ce4e14b0c35"}
Oct 30 14:32:59.527896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.527613    4530 scope.go:122] "RemoveContainer" containerID="14b2a068f659197422cca734f2d5a9cf8ae07da516bad24d189aabe2f4be8445"
Oct 30 14:32:59.616616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.616593    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:32:59.635344 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:32:59.635313    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8240e208_5f61_45a4_b0e2_e6298ce7ab8e.slice/crio-6fb4bdf03a61671046da5815f808399b26d72e253126118917c7dee34a210562 WatchSource:0}: Error finding container 6fb4bdf03a61671046da5815f808399b26d72e253126118917c7dee34a210562: Status 404 returned error can't find the container with id 6fb4bdf03a61671046da5815f808399b26d72e253126118917c7dee34a210562
Oct 30 14:32:59.638226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.638204    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"]
Oct 30 14:32:59.691015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.690985    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-podinfo\") pod \"a83aae0b-4976-4a91-937c-c1e3dda7f151\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") "
Oct 30 14:32:59.691168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.691048    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-data\") pod \"a83aae0b-4976-4a91-937c-c1e3dda7f151\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") "
Oct 30 14:32:59.691168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.691104    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-credential-socket\") pod \"a83aae0b-4976-4a91-937c-c1e3dda7f151\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") "
Oct 30 14:32:59.691168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.691139    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83aae0b-4976-4a91-937c-c1e3dda7f151-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83aae0b-4976-4a91-937c-c1e3dda7f151-istiod-ca-cert\") pod \"a83aae0b-4976-4a91-937c-c1e3dda7f151\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") "
Oct 30 14:32:59.691359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.691217    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-socket\") pod \"a83aae0b-4976-4a91-937c-c1e3dda7f151\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") "
Oct 30 14:32:59.691359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.691339    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-envoy\") pod \"a83aae0b-4976-4a91-937c-c1e3dda7f151\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") "
Oct 30 14:32:59.691475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.691388    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-token\" (UniqueName: \"kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-token\") pod \"a83aae0b-4976-4a91-937c-c1e3dda7f151\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") "
Oct 30 14:32:59.691475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.691440    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-data" pod "a83aae0b-4976-4a91-937c-c1e3dda7f151" (UID: "a83aae0b-4976-4a91-937c-c1e3dda7f151"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:59.691623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.691475    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-kube-api-access-v88zt\" (UniqueName: \"kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-kube-api-access-v88zt\") pod \"a83aae0b-4976-4a91-937c-c1e3dda7f151\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") "
Oct 30 14:32:59.691623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.691510    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-certs\") pod \"a83aae0b-4976-4a91-937c-c1e3dda7f151\" (UID: \"a83aae0b-4976-4a91-937c-c1e3dda7f151\") "
Oct 30 14:32:59.691747 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.691663    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-socket" pod "a83aae0b-4976-4a91-937c-c1e3dda7f151" (UID: "a83aae0b-4976-4a91-937c-c1e3dda7f151"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:59.691996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.691969    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-certs" pod "a83aae0b-4976-4a91-937c-c1e3dda7f151" (UID: "a83aae0b-4976-4a91-937c-c1e3dda7f151"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:59.692208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.692186    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-credential-socket" pod "a83aae0b-4976-4a91-937c-c1e3dda7f151" (UID: "a83aae0b-4976-4a91-937c-c1e3dda7f151"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:59.692331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.692211    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:59.692331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.692238    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:59.692331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.692293    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:59.692331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.692257    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83aae0b-4976-4a91-937c-c1e3dda7f151-istiod-ca-cert" pod "a83aae0b-4976-4a91-937c-c1e3dda7f151" (UID: "a83aae0b-4976-4a91-937c-c1e3dda7f151"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:32:59.694256 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.694219    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-envoy" pod "a83aae0b-4976-4a91-937c-c1e3dda7f151" (UID: "a83aae0b-4976-4a91-937c-c1e3dda7f151"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:32:59.694437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.694412    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-token" pod "a83aae0b-4976-4a91-937c-c1e3dda7f151" (UID: "a83aae0b-4976-4a91-937c-c1e3dda7f151"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:32:59.696730 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.696699    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-kube-api-access-v88zt" pod "a83aae0b-4976-4a91-937c-c1e3dda7f151" (UID: "a83aae0b-4976-4a91-937c-c1e3dda7f151"). InnerVolumeSpecName "kube-api-access-v88zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:32:59.699505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.699482    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-podinfo" pod "a83aae0b-4976-4a91-937c-c1e3dda7f151" (UID: "a83aae0b-4976-4a91-937c-c1e3dda7f151"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:32:59.793186 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.793147    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:59.793186 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.793179    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:59.793402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.793193    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v88zt\" (UniqueName: \"kubernetes.io/projected/a83aae0b-4976-4a91-937c-c1e3dda7f151-kube-api-access-v88zt\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:59.793402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.793208    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83aae0b-4976-4a91-937c-c1e3dda7f151-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:59.793402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.793225    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83aae0b-4976-4a91-937c-c1e3dda7f151-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:32:59.793402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:32:59.793237    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83aae0b-4976-4a91-937c-c1e3dda7f151-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:00.535961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:00.535877    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"
Oct 30 14:33:00.535961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:00.535930    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6" event={"ID":"a83aae0b-4976-4a91-937c-c1e3dda7f151","Type":"ContainerDied","Data":"7703fbb71d345a426fc2ab0c36297ccaf3927a9dc2603ed2362e8b298a9fae24"}
Oct 30 14:33:00.536458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:00.535993    4530 scope.go:122] "RemoveContainer" containerID="ee7fc65c1cc0cede66d222bcd7d1f5418082bd89ad06834169913ce4e14b0c35"
Oct 30 14:33:00.538074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:00.538044    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c" event={"ID":"8240e208-5f61-45a4-b0e2-e6298ce7ab8e","Type":"ContainerStarted","Data":"3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692"}
Oct 30 14:33:00.538214 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:00.538101    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c" event={"ID":"8240e208-5f61-45a4-b0e2-e6298ce7ab8e","Type":"ContainerStarted","Data":"6fb4bdf03a61671046da5815f808399b26d72e253126118917c7dee34a210562"}
Oct 30 14:33:00.538214 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:00.538131    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"
Oct 30 14:33:00.540173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:00.540148    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g" event={"ID":"52ec6eeb-78f9-4100-8390-c9688a8f94dc","Type":"ContainerStarted","Data":"38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774"}
Oct 30 14:33:00.540270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:00.540196    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"
Oct 30 14:33:00.614915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:00.614888    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"]
Oct 30 14:33:00.628742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:00.628716    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-rq7j6"]
Oct 30 14:33:00.659606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:00.659560    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g" podStartSLOduration=2.6595335909999998 podStartE2EDuration="2.659533591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:00.658670766 +0000 UTC m=+2645.690569704" watchObservedRunningTime="2026-10-30 14:33:00.659533591 +0000 UTC m=+2645.691432529"
Oct 30 14:33:00.706215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:00.706165    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c" podStartSLOduration=2.706151802 podStartE2EDuration="2.706151802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:00.704173113 +0000 UTC m=+2645.736072077" watchObservedRunningTime="2026-10-30 14:33:00.706151802 +0000 UTC m=+2645.738050740"
Oct 30 14:33:01.116208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:01.116177    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" path="/var/lib/kubelet/pods/a83aae0b-4976-4a91-937c-c1e3dda7f151/volumes"
Oct 30 14:33:02.139500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:02.139465    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:33:02.152644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:02.152616    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:33:02.570707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:02.570676    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:33:03.253590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.253520    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"]
Oct 30 14:33:03.254072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.253764    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt" podUID="1152d262-34c5-4c25-83d4-280abb53f12d" containerName="maas-api" containerID="cri-o://97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9" gracePeriod=30
Oct 30 14:33:03.601566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.600451    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:33:03.618263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.617398    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"
Oct 30 14:33:03.618263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.617647    4530 generic.go:356] "Generic (PLEG): container finished" podID="1152d262-34c5-4c25-83d4-280abb53f12d" containerID="97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9" exitCode=0
Oct 30 14:33:03.618651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.618626    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt" event={"ID":"1152d262-34c5-4c25-83d4-280abb53f12d","Type":"ContainerDied","Data":"97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9"}
Oct 30 14:33:03.618771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.618678    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt" event={"ID":"1152d262-34c5-4c25-83d4-280abb53f12d","Type":"ContainerDied","Data":"ce8f66ebd21ad90b235cf0f5b57c9f37f15a043eae64c6cc17c61426bbff703a"}
Oct 30 14:33:03.618771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.618708    4530 scope.go:122] "RemoveContainer" containerID="97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9"
Oct 30 14:33:03.632559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.632521    4530 scope.go:122] "RemoveContainer" containerID="97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9"
Oct 30 14:33:03.632860 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:33:03.632841    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9\": container with ID starting with 97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9 not found: ID does not exist" containerID="97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9"
Oct 30 14:33:03.632946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.632869    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9"} err="failed to get container status \"97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9\": rpc error: code = NotFound desc = could not find container \"97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9\": container with ID starting with 97b35c52cfd29326f2fd56c6662402b13e1b873b34b9ab96c9511402155294a9 not found: ID does not exist"
Oct 30 14:33:03.668120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.667940    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-metrics-certs\") pod \"1152d262-34c5-4c25-83d4-280abb53f12d\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") "
Oct 30 14:33:03.668120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.668104    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-maas-api-tls\") pod \"1152d262-34c5-4c25-83d4-280abb53f12d\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") "
Oct 30 14:33:03.668383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.668179    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1152d262-34c5-4c25-83d4-280abb53f12d-kube-api-access-5m8k6\" (UniqueName: \"kubernetes.io/projected/1152d262-34c5-4c25-83d4-280abb53f12d-kube-api-access-5m8k6\") pod \"1152d262-34c5-4c25-83d4-280abb53f12d\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") "
Oct 30 14:33:03.668446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.668422    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1152d262-34c5-4c25-83d4-280abb53f12d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1152d262-34c5-4c25-83d4-280abb53f12d-openshift-service-ca\") pod \"1152d262-34c5-4c25-83d4-280abb53f12d\" (UID: \"1152d262-34c5-4c25-83d4-280abb53f12d\") "
Oct 30 14:33:03.687527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.687487    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1152d262-34c5-4c25-83d4-280abb53f12d-openshift-service-ca" pod "1152d262-34c5-4c25-83d4-280abb53f12d" (UID: "1152d262-34c5-4c25-83d4-280abb53f12d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:33:03.770849 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.768793    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-metrics-certs" pod "1152d262-34c5-4c25-83d4-280abb53f12d" (UID: "1152d262-34c5-4c25-83d4-280abb53f12d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:33:03.784578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.773552    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1152d262-34c5-4c25-83d4-280abb53f12d-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:03.784578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.773589    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:03.784578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.781020    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1152d262-34c5-4c25-83d4-280abb53f12d-kube-api-access-5m8k6" pod "1152d262-34c5-4c25-83d4-280abb53f12d" (UID: "1152d262-34c5-4c25-83d4-280abb53f12d"). InnerVolumeSpecName "kube-api-access-5m8k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:33:03.784578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.782197    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-maas-api-tls" pod "1152d262-34c5-4c25-83d4-280abb53f12d" (UID: "1152d262-34c5-4c25-83d4-280abb53f12d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:33:03.874590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.874521    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1152d262-34c5-4c25-83d4-280abb53f12d-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:03.874590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.874590    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m8k6\" (UniqueName: \"kubernetes.io/projected/1152d262-34c5-4c25-83d4-280abb53f12d-kube-api-access-5m8k6\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:03.975414 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.975381    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"]
Oct 30 14:33:03.986926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:03.986890    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b9c4b4-5584584d8b-5kdpt"]
Oct 30 14:33:05.122919 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:05.122875    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1152d262-34c5-4c25-83d4-280abb53f12d" path="/var/lib/kubelet/pods/1152d262-34c5-4c25-83d4-280abb53f12d/volumes"
Oct 30 14:33:06.337921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:06.337889    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"]
Oct 30 14:33:11.367469 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:33:11.367396    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:33:11.556203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:11.556164    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"
Oct 30 14:33:11.556351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:11.556251    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"
Oct 30 14:33:12.277396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:12.277360    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"]
Oct 30 14:33:12.729698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:12.729635    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy" containerID="cri-o://076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105" gracePeriod=30
Oct 30 14:33:14.380144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:14.380105    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"]
Oct 30 14:33:14.380685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:14.380316    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" podUID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" containerName="istio-proxy" containerID="cri-o://05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b" gracePeriod=30
Oct 30 14:33:14.761597 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:14.761486    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:33:16.634810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:16.634768    4530 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:33:16.635261 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:16.634860    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" podUID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:33:17.995891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:17.995829    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:33:18.092769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.092734    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-podinfo\") pod \"e271438d-7e48-47fe-887b-d3ba07d2a875\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") "
Oct 30 14:33:18.092987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.092811    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-credential-socket\") pod \"e271438d-7e48-47fe-887b-d3ba07d2a875\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") "
Oct 30 14:33:18.093067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.093010    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-socket\") pod \"e271438d-7e48-47fe-887b-d3ba07d2a875\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") "
Oct 30 14:33:18.093134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.093065    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-envoy\") pod \"e271438d-7e48-47fe-887b-d3ba07d2a875\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") "
Oct 30 14:33:18.093134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.093095    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e271438d-7e48-47fe-887b-d3ba07d2a875-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e271438d-7e48-47fe-887b-d3ba07d2a875-istiod-ca-cert\") pod \"e271438d-7e48-47fe-887b-d3ba07d2a875\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") "
Oct 30 14:33:18.093134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.093129    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-token\" (UniqueName: \"kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-token\") pod \"e271438d-7e48-47fe-887b-d3ba07d2a875\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") "
Oct 30 14:33:18.093298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.093169    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-data\") pod \"e271438d-7e48-47fe-887b-d3ba07d2a875\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") "
Oct 30 14:33:18.093298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.093283    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-kube-api-access-9pkhm\" (UniqueName: \"kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-kube-api-access-9pkhm\") pod \"e271438d-7e48-47fe-887b-d3ba07d2a875\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") "
Oct 30 14:33:18.093406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.093295    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-credential-socket" pod "e271438d-7e48-47fe-887b-d3ba07d2a875" (UID: "e271438d-7e48-47fe-887b-d3ba07d2a875"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:18.093406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.093315    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-certs\") pod \"e271438d-7e48-47fe-887b-d3ba07d2a875\" (UID: \"e271438d-7e48-47fe-887b-d3ba07d2a875\") "
Oct 30 14:33:18.093651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.093631    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e271438d-7e48-47fe-887b-d3ba07d2a875-istiod-ca-cert" pod "e271438d-7e48-47fe-887b-d3ba07d2a875" (UID: "e271438d-7e48-47fe-887b-d3ba07d2a875"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:33:18.094416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.093868    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-certs" pod "e271438d-7e48-47fe-887b-d3ba07d2a875" (UID: "e271438d-7e48-47fe-887b-d3ba07d2a875"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:18.094416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.094139    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e271438d-7e48-47fe-887b-d3ba07d2a875-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:18.094416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.094160    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:18.094416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.094173    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:18.094416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.094310    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-socket" pod "e271438d-7e48-47fe-887b-d3ba07d2a875" (UID: "e271438d-7e48-47fe-887b-d3ba07d2a875"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:18.095150 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.095112    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-data" pod "e271438d-7e48-47fe-887b-d3ba07d2a875" (UID: "e271438d-7e48-47fe-887b-d3ba07d2a875"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:18.096509 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.096480    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-podinfo" pod "e271438d-7e48-47fe-887b-d3ba07d2a875" (UID: "e271438d-7e48-47fe-887b-d3ba07d2a875"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:33:18.096646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.096608    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-kube-api-access-9pkhm" pod "e271438d-7e48-47fe-887b-d3ba07d2a875" (UID: "e271438d-7e48-47fe-887b-d3ba07d2a875"). InnerVolumeSpecName "kube-api-access-9pkhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:33:18.097137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.097106    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-token" pod "e271438d-7e48-47fe-887b-d3ba07d2a875" (UID: "e271438d-7e48-47fe-887b-d3ba07d2a875"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:33:18.097215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.097171    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-envoy" pod "e271438d-7e48-47fe-887b-d3ba07d2a875" (UID: "e271438d-7e48-47fe-887b-d3ba07d2a875"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:18.195084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.195050    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:18.195084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.195079    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:18.195084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.195087    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:18.195297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.195096    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pkhm\" (UniqueName: \"kubernetes.io/projected/e271438d-7e48-47fe-887b-d3ba07d2a875-kube-api-access-9pkhm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:18.195297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.195105    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e271438d-7e48-47fe-887b-d3ba07d2a875-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:18.195297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.195113    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271438d-7e48-47fe-887b-d3ba07d2a875-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:18.790986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.790946    4530 generic.go:356] "Generic (PLEG): container finished" podID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerID="076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105" exitCode=0
Oct 30 14:33:18.791171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.791002    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"
Oct 30 14:33:18.791171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.791029    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" event={"ID":"e271438d-7e48-47fe-887b-d3ba07d2a875","Type":"ContainerDied","Data":"076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105"}
Oct 30 14:33:18.791171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.791096    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl" event={"ID":"e271438d-7e48-47fe-887b-d3ba07d2a875","Type":"ContainerDied","Data":"c2fcb01f4da372d3fcff83773a552b24f87df2949426ca14615794d7973263d8"}
Oct 30 14:33:18.791171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.791127    4530 scope.go:122] "RemoveContainer" containerID="076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105"
Oct 30 14:33:18.808175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.808151    4530 scope.go:122] "RemoveContainer" containerID="5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a"
Oct 30 14:33:18.819125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.819102    4530 scope.go:122] "RemoveContainer" containerID="076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105"
Oct 30 14:33:18.819401 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:33:18.819381    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105\": container with ID starting with 076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105 not found: ID does not exist" containerID="076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105"
Oct 30 14:33:18.819480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.819410    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105"} err="failed to get container status \"076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105\": rpc error: code = NotFound desc = could not find container \"076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105\": container with ID starting with 076c15f8b2d7fb74848339978a56850651b11e566c5659c7a171112a7d63f105 not found: ID does not exist"
Oct 30 14:33:18.819480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.819431    4530 scope.go:122] "RemoveContainer" containerID="5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a"
Oct 30 14:33:18.819729 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:33:18.819705    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a\": container with ID starting with 5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a not found: ID does not exist" containerID="5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a"
Oct 30 14:33:18.819782 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.819738    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a"} err="failed to get container status \"5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a\": rpc error: code = NotFound desc = could not find container \"5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a\": container with ID starting with 5ad68f60153b13a5a1e4a916a6a657112914157d8f82f602b95e0284e6af839a not found: ID does not exist"
Oct 30 14:33:18.868253 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.868230    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"]
Oct 30 14:33:18.882425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:18.882400    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-9dhnl"]
Oct 30 14:33:19.127746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.127702    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" path="/var/lib/kubelet/pods/e271438d-7e48-47fe-887b-d3ba07d2a875/volumes"
Oct 30 14:33:19.644213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.644158    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:33:19.801356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.801319    4530 generic.go:356] "Generic (PLEG): container finished" podID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" containerID="05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b" exitCode=0
Oct 30 14:33:19.801532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.801376    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"
Oct 30 14:33:19.801532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.801397    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" event={"ID":"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa","Type":"ContainerDied","Data":"05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b"}
Oct 30 14:33:19.801532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.801444    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg" event={"ID":"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa","Type":"ContainerDied","Data":"8bc90fd620d47086765900a8e8fe51555a6b0f49ec4cb4563fa4949db0de12d2"}
Oct 30 14:33:19.801532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.801471    4530 scope.go:122] "RemoveContainer" containerID="05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b"
Oct 30 14:33:19.815117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.815089    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istiod-ca-cert\") pod \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") "
Oct 30 14:33:19.815274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.815145    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-kube-api-access-8kr5t\" (UniqueName: \"kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-kube-api-access-8kr5t\") pod \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") "
Oct 30 14:33:19.815274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.815179    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-podinfo\") pod \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") "
Oct 30 14:33:19.815381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.815277    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-credential-socket\") pod \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") "
Oct 30 14:33:19.815381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.815340    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-data\") pod \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") "
Oct 30 14:33:19.815509 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.815487    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istiod-ca-cert" pod "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" (UID: "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:33:19.815611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.815501    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-socket\") pod \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") "
Oct 30 14:33:19.815611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.815563    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-envoy\") pod \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") "
Oct 30 14:33:19.815611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.815597    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-certs\") pod \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") "
Oct 30 14:33:19.815777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.815721    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-socket" pod "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" (UID: "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:19.815828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.815777    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-token\" (UniqueName: \"kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-token\") pod \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\" (UID: \"3753a7f2-b5bb-465d-85c1-3e2f86b57cfa\") "
Oct 30 14:33:19.816084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.816059    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-certs" pod "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" (UID: "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:19.816660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.816493    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-data" pod "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" (UID: "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:19.816660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.816503    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:19.816660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.816530    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:19.816660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.816557    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:19.817185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.817155    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-credential-socket" pod "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" (UID: "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:19.818625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.818595    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-kube-api-access-8kr5t" pod "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" (UID: "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa"). InnerVolumeSpecName "kube-api-access-8kr5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:33:19.818739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.818726    4530 scope.go:122] "RemoveContainer" containerID="05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b"
Oct 30 14:33:19.819442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.819419    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-podinfo" pod "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" (UID: "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:33:19.819599 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:33:19.819428    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b\": container with ID starting with 05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b not found: ID does not exist" containerID="05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b"
Oct 30 14:33:19.819707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.819611    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b"} err="failed to get container status \"05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b\": rpc error: code = NotFound desc = could not find container \"05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b\": container with ID starting with 05dea7d3d9d9532944825b3daa3a9184e4437ad15c99029f7708bf1648ee339b not found: ID does not exist"
Oct 30 14:33:19.819707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.819655    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-envoy" pod "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" (UID: "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:19.819707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.819660    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-token" pod "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" (UID: "3753a7f2-b5bb-465d-85c1-3e2f86b57cfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:33:19.918383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.918316    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:19.918383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.918344    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:19.918383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.918354    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:19.918383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.918363    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:19.918383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.918371    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kr5t\" (UniqueName: \"kubernetes.io/projected/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-kube-api-access-8kr5t\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:19.918383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:19.918379    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:20.156701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:20.156668    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"]
Oct 30 14:33:20.171174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:20.171111    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-kwzdg"]
Oct 30 14:33:21.118021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.117985    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" path="/var/lib/kubelet/pods/3753a7f2-b5bb-465d-85c1-3e2f86b57cfa/volumes"
Oct 30 14:33:21.358730 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.358697    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w" podUID="47456e25-a0e3-4610-b455-4dd8f428ebf8" containerName="main" containerID="cri-o://763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf" gracePeriod=45
Oct 30 14:33:21.622080 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.622053    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:33:21.635815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.635753    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47456e25-a0e3-4610-b455-4dd8f428ebf8-kube-api-access-xrmnx\" (UniqueName: \"kubernetes.io/projected/47456e25-a0e3-4610-b455-4dd8f428ebf8-kube-api-access-xrmnx\") pod \"47456e25-a0e3-4610-b455-4dd8f428ebf8\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") "
Oct 30 14:33:21.639809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.635803    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-model-cache\") pod \"47456e25-a0e3-4610-b455-4dd8f428ebf8\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") "
Oct 30 14:33:21.639809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.635934    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47456e25-a0e3-4610-b455-4dd8f428ebf8-tls-certs\" (UniqueName: \"kubernetes.io/secret/47456e25-a0e3-4610-b455-4dd8f428ebf8-tls-certs\") pod \"47456e25-a0e3-4610-b455-4dd8f428ebf8\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") "
Oct 30 14:33:21.639809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.636044    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-dshm\") pod \"47456e25-a0e3-4610-b455-4dd8f428ebf8\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") "
Oct 30 14:33:21.639809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.636073    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-tmp-dir\") pod \"47456e25-a0e3-4610-b455-4dd8f428ebf8\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") "
Oct 30 14:33:21.639809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.636077    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-model-cache" pod "47456e25-a0e3-4610-b455-4dd8f428ebf8" (UID: "47456e25-a0e3-4610-b455-4dd8f428ebf8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:21.639809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.636305    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-home\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-home\") pod \"47456e25-a0e3-4610-b455-4dd8f428ebf8\" (UID: \"47456e25-a0e3-4610-b455-4dd8f428ebf8\") "
Oct 30 14:33:21.639809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.636850    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:21.639809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.637088    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-home" pod "47456e25-a0e3-4610-b455-4dd8f428ebf8" (UID: "47456e25-a0e3-4610-b455-4dd8f428ebf8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:21.639809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.637010    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-tmp-dir" pod "47456e25-a0e3-4610-b455-4dd8f428ebf8" (UID: "47456e25-a0e3-4610-b455-4dd8f428ebf8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:21.640442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.640420    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-dshm" pod "47456e25-a0e3-4610-b455-4dd8f428ebf8" (UID: "47456e25-a0e3-4610-b455-4dd8f428ebf8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:21.641400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.641376    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47456e25-a0e3-4610-b455-4dd8f428ebf8-tls-certs" pod "47456e25-a0e3-4610-b455-4dd8f428ebf8" (UID: "47456e25-a0e3-4610-b455-4dd8f428ebf8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:33:21.641566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.641534    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47456e25-a0e3-4610-b455-4dd8f428ebf8-kube-api-access-xrmnx" pod "47456e25-a0e3-4610-b455-4dd8f428ebf8" (UID: "47456e25-a0e3-4610-b455-4dd8f428ebf8"). InnerVolumeSpecName "kube-api-access-xrmnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:33:21.668370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.668342    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"]
Oct 30 14:33:21.668587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.668562    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy" containerID="cri-o://e1fd18b51491b08c1d1c5f5e0ff01aa752623fd7f1330ed80fc15d483695a31a" gracePeriod=30
Oct 30 14:33:21.737805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.737774    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:21.737805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.737801    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:21.737805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.737812    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47456e25-a0e3-4610-b455-4dd8f428ebf8-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:21.738038 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.737821    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrmnx\" (UniqueName: \"kubernetes.io/projected/47456e25-a0e3-4610-b455-4dd8f428ebf8-kube-api-access-xrmnx\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:21.738038 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.737831    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47456e25-a0e3-4610-b455-4dd8f428ebf8-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:21.862304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.862267    4530 generic.go:356] "Generic (PLEG): container finished" podID="47456e25-a0e3-4610-b455-4dd8f428ebf8" containerID="763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf" exitCode=0
Oct 30 14:33:21.862498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.862483    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w" event={"ID":"47456e25-a0e3-4610-b455-4dd8f428ebf8","Type":"ContainerDied","Data":"763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf"}
Oct 30 14:33:21.862594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.862524    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w" event={"ID":"47456e25-a0e3-4610-b455-4dd8f428ebf8","Type":"ContainerDied","Data":"d9a29888b9d83c43eb12f9479e2085b469c496ea2001138e56c18b680f01f2bc"}
Oct 30 14:33:21.862594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.862566    4530 scope.go:122] "RemoveContainer" containerID="763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf"
Oct 30 14:33:21.862727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.862713    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"
Oct 30 14:33:21.892061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.892033    4530 scope.go:122] "RemoveContainer" containerID="763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf"
Oct 30 14:33:21.892534 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:33:21.892511    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf\": container with ID starting with 763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf not found: ID does not exist" containerID="763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf"
Oct 30 14:33:21.892639 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.892601    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf"} err="failed to get container status \"763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf\": rpc error: code = NotFound desc = could not find container \"763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf\": container with ID starting with 763e64e7a7738bb2a8b808c0ebeb3e569d7ae9fa3148c39d8451fdca5e134ecf not found: ID does not exist"
Oct 30 14:33:21.928569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.928476    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"]
Oct 30 14:33:21.934208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:21.934182    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-019e5b/test-model-019e5b-kserve-64b9ff6587-2tb5w"]
Oct 30 14:33:23.124982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:23.124947    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47456e25-a0e3-4610-b455-4dd8f428ebf8" path="/var/lib/kubelet/pods/47456e25-a0e3-4610-b455-4dd8f428ebf8/volumes"
Oct 30 14:33:25.856110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:25.856072    4530 patch_prober.go:29] interesting pod/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:33:25.856501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:25.856143    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:33:26.901903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:26.901864    4530 generic.go:356] "Generic (PLEG): container finished" podID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerID="e1fd18b51491b08c1d1c5f5e0ff01aa752623fd7f1330ed80fc15d483695a31a" exitCode=0
Oct 30 14:33:26.902387 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:26.901932    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" event={"ID":"06147d62-09b3-4e0e-bb3a-a8fad254a8dd","Type":"ContainerDied","Data":"e1fd18b51491b08c1d1c5f5e0ff01aa752623fd7f1330ed80fc15d483695a31a"}
Oct 30 14:33:26.902387 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:26.902010    4530 scope.go:122] "RemoveContainer" containerID="e7460e84dd32a353e5f78cbaa9bb41be362e6514c17d918359dea4465abc2dfc"
Oct 30 14:33:26.945610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:26.945559    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:33:27.021482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.021453    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-envoy\") pod \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") "
Oct 30 14:33:27.021689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.021579    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-certs\") pod \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") "
Oct 30 14:33:27.021689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.021626    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istiod-ca-cert\") pod \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") "
Oct 30 14:33:27.021689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.021656    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-credential-socket\") pod \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") "
Oct 30 14:33:27.021891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.021714    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-token\" (UniqueName: \"kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-token\") pod \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") "
Oct 30 14:33:27.021891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.021763    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-kube-api-access-wt755\" (UniqueName: \"kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-kube-api-access-wt755\") pod \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") "
Oct 30 14:33:27.021891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.021792    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-data\") pod \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") "
Oct 30 14:33:27.021891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.021876    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-socket\") pod \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") "
Oct 30 14:33:27.022277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.022016    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-podinfo\") pod \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\" (UID: \"06147d62-09b3-4e0e-bb3a-a8fad254a8dd\") "
Oct 30 14:33:27.023351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.023321    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-credential-socket" pod "06147d62-09b3-4e0e-bb3a-a8fad254a8dd" (UID: "06147d62-09b3-4e0e-bb3a-a8fad254a8dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:27.024244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.024163    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-certs" pod "06147d62-09b3-4e0e-bb3a-a8fad254a8dd" (UID: "06147d62-09b3-4e0e-bb3a-a8fad254a8dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:27.024434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.024393    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-socket" pod "06147d62-09b3-4e0e-bb3a-a8fad254a8dd" (UID: "06147d62-09b3-4e0e-bb3a-a8fad254a8dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:27.024613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.024535    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istiod-ca-cert" pod "06147d62-09b3-4e0e-bb3a-a8fad254a8dd" (UID: "06147d62-09b3-4e0e-bb3a-a8fad254a8dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:33:27.024743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.024360    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-data" pod "06147d62-09b3-4e0e-bb3a-a8fad254a8dd" (UID: "06147d62-09b3-4e0e-bb3a-a8fad254a8dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:27.026469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.026447    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-podinfo" pod "06147d62-09b3-4e0e-bb3a-a8fad254a8dd" (UID: "06147d62-09b3-4e0e-bb3a-a8fad254a8dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:33:27.026608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.026501    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-kube-api-access-wt755" pod "06147d62-09b3-4e0e-bb3a-a8fad254a8dd" (UID: "06147d62-09b3-4e0e-bb3a-a8fad254a8dd"). InnerVolumeSpecName "kube-api-access-wt755". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:33:27.027801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.027779    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-token" pod "06147d62-09b3-4e0e-bb3a-a8fad254a8dd" (UID: "06147d62-09b3-4e0e-bb3a-a8fad254a8dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:33:27.031470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.031446    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-envoy" pod "06147d62-09b3-4e0e-bb3a-a8fad254a8dd" (UID: "06147d62-09b3-4e0e-bb3a-a8fad254a8dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:27.123769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.123745    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:27.123769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.123770    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt755\" (UniqueName: \"kubernetes.io/projected/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-kube-api-access-wt755\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:27.123926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.123779    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:27.123926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.123789    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:27.123926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.123797    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:27.123926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.123806    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:27.123926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.123815    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:27.123926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.123822    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:27.123926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.123829    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06147d62-09b3-4e0e-bb3a-a8fad254a8dd-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:27.909460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.909427    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"
Oct 30 14:33:27.909944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.909468    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk" event={"ID":"06147d62-09b3-4e0e-bb3a-a8fad254a8dd","Type":"ContainerDied","Data":"3a64bcb679814b74569a24414146882102cad22608cc9bf29e29a6c2b09b6517"}
Oct 30 14:33:27.909944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.909529    4530 scope.go:122] "RemoveContainer" containerID="e1fd18b51491b08c1d1c5f5e0ff01aa752623fd7f1330ed80fc15d483695a31a"
Oct 30 14:33:27.995768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:27.995726    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"]
Oct 30 14:33:28.002835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:28.002808    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b9c4b4-openshift-default-756965fdd5-jz5hk"]
Oct 30 14:33:29.115713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:29.115677    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" path="/var/lib/kubelet/pods/06147d62-09b3-4e0e-bb3a-a8fad254a8dd/volumes"
Oct 30 14:33:30.037179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.037140    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"]
Oct 30 14:33:30.037417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.037390    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657" podUID="9b35d7cc-e2d8-4595-9d9b-06b4be0165db" containerName="maas-api" containerID="cri-o://0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3" gracePeriod=30
Oct 30 14:33:30.308051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.308023    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:33:30.359996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.359957    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-openshift-service-ca\") pod \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") "
Oct 30 14:33:30.360183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.360016    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-maas-api-tls\") pod \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") "
Oct 30 14:33:30.360247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.360214    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-metrics-certs\") pod \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") "
Oct 30 14:33:30.360377 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.360349    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-openshift-service-ca" pod "9b35d7cc-e2d8-4595-9d9b-06b4be0165db" (UID: "9b35d7cc-e2d8-4595-9d9b-06b4be0165db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:33:30.360696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.360667    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-kube-api-access-vb7jr\" (UniqueName: \"kubernetes.io/projected/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-kube-api-access-vb7jr\") pod \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\" (UID: \"9b35d7cc-e2d8-4595-9d9b-06b4be0165db\") "
Oct 30 14:33:30.361298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.361275    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:30.362347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.362323    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-maas-api-tls" pod "9b35d7cc-e2d8-4595-9d9b-06b4be0165db" (UID: "9b35d7cc-e2d8-4595-9d9b-06b4be0165db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:33:30.362757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.362735    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-metrics-certs" pod "9b35d7cc-e2d8-4595-9d9b-06b4be0165db" (UID: "9b35d7cc-e2d8-4595-9d9b-06b4be0165db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:33:30.363033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.363002    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-kube-api-access-vb7jr" pod "9b35d7cc-e2d8-4595-9d9b-06b4be0165db" (UID: "9b35d7cc-e2d8-4595-9d9b-06b4be0165db"). InnerVolumeSpecName "kube-api-access-vb7jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:33:30.462818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.462784    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:30.462818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.462812    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb7jr\" (UniqueName: \"kubernetes.io/projected/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-kube-api-access-vb7jr\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:30.462818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.462822    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b35d7cc-e2d8-4595-9d9b-06b4be0165db-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:30.638663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.638579    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"]
Oct 30 14:33:30.938917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.938823    4530 generic.go:356] "Generic (PLEG): container finished" podID="9b35d7cc-e2d8-4595-9d9b-06b4be0165db" containerID="0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3" exitCode=0
Oct 30 14:33:30.938917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.938868    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"
Oct 30 14:33:30.938917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.938889    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657" event={"ID":"9b35d7cc-e2d8-4595-9d9b-06b4be0165db","Type":"ContainerDied","Data":"0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3"}
Oct 30 14:33:30.939241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.938948    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657" event={"ID":"9b35d7cc-e2d8-4595-9d9b-06b4be0165db","Type":"ContainerDied","Data":"e083484b7783c0fbb7c35b552f92a235c94c374216e547269804cefe5762ee2e"}
Oct 30 14:33:30.939241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.938985    4530 scope.go:122] "RemoveContainer" containerID="0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3"
Oct 30 14:33:30.955306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.955280    4530 scope.go:122] "RemoveContainer" containerID="0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3"
Oct 30 14:33:30.955695 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:33:30.955673    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3\": container with ID starting with 0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3 not found: ID does not exist" containerID="0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3"
Oct 30 14:33:30.955781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:30.955703    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3"} err="failed to get container status \"0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3\": rpc error: code = NotFound desc = could not find container \"0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3\": container with ID starting with 0d9b7e4aca385ea0490645ab42ed169737fb73454463b5fd98e309e9851547d3 not found: ID does not exist"
Oct 30 14:33:31.004069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:31.004032    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"]
Oct 30 14:33:31.011161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:31.011134    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b1c15-c544d55f8-zs657"]
Oct 30 14:33:31.115555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:31.115512    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b35d7cc-e2d8-4595-9d9b-06b4be0165db" path="/var/lib/kubelet/pods/9b35d7cc-e2d8-4595-9d9b-06b4be0165db/volumes"
Oct 30 14:33:32.119511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.119473    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"]
Oct 30 14:33:32.120359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120338    4530 cpu_manager.go:414] "containerMap: removing container" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy"
Oct 30 14:33:32.120359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120357    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120368    4530 cpu_manager.go:414] "containerMap: removing container" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120374    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120381    4530 cpu_manager.go:414] "containerMap: removing container" podUID="47456e25-a0e3-4610-b455-4dd8f428ebf8" containerName="main"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120389    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47456e25-a0e3-4610-b455-4dd8f428ebf8" containerName="main"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120410    4530 cpu_manager.go:414] "containerMap: removing container" podUID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" containerName="istio-proxy"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120419    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" containerName="istio-proxy"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120450    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9b35d7cc-e2d8-4595-9d9b-06b4be0165db" containerName="maas-api"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120457    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b35d7cc-e2d8-4595-9d9b-06b4be0165db" containerName="maas-api"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120466    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1152d262-34c5-4c25-83d4-280abb53f12d" containerName="maas-api"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120473    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1152d262-34c5-4c25-83d4-280abb53f12d" containerName="maas-api"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120507    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120513    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy"
Oct 30 14:33:32.120530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120526    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy"
Oct 30 14:33:32.121265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120534    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy"
Oct 30 14:33:32.121265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120726    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy"
Oct 30 14:33:32.121265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120739    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e271438d-7e48-47fe-887b-d3ba07d2a875" containerName="istio-proxy"
Oct 30 14:33:32.121265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120754    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy"
Oct 30 14:33:32.121265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120762    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy"
Oct 30 14:33:32.121265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120770    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="47456e25-a0e3-4610-b455-4dd8f428ebf8" containerName="main"
Oct 30 14:33:32.121265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120783    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="3753a7f2-b5bb-465d-85c1-3e2f86b57cfa" containerName="istio-proxy"
Oct 30 14:33:32.121265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120792    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy"
Oct 30 14:33:32.121265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120803    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b35d7cc-e2d8-4595-9d9b-06b4be0165db" containerName="maas-api"
Oct 30 14:33:32.121265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.120811    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1152d262-34c5-4c25-83d4-280abb53f12d" containerName="maas-api"
Oct 30 14:33:32.125556 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.125517    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.142272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.142245    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"]
Oct 30 14:33:32.152829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.152802    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-75443e\"/\"default-dockercfg-smxvj\""
Oct 30 14:33:32.153027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.152995    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-75443e\"/\"auth-test-model-75443e-kserve-self-signed-certs\""
Oct 30 14:33:32.160015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.159993    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-75443e\"/\"openshift-service-ca.crt\""
Oct 30 14:33:32.161155 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.161135    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-75443e\"/\"kube-root-ca.crt\""
Oct 30 14:33:32.203381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.203353    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-dshm\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.203509 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.203481    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb89111d-7a71-4de1-994a-7470dc21d733-tls-certs\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.203601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.203575    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-home\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.203694 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.203674    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb26w\" (UniqueName: \"kubernetes.io/projected/eb89111d-7a71-4de1-994a-7470dc21d733-kube-api-access-xb26w\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.203774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.203760    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-tmp-dir\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.203838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.203830    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-model-cache\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.305225 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.305197    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-model-cache\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.305369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.305248    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-dshm\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.305369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.305293    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb89111d-7a71-4de1-994a-7470dc21d733-tls-certs\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.305369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.305318    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-home\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.305369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.305360    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb26w\" (UniqueName: \"kubernetes.io/projected/eb89111d-7a71-4de1-994a-7470dc21d733-kube-api-access-xb26w\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.305718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.305427    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-tmp-dir\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.305776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.305713    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-model-cache\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.305776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.305756    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-home\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.305857 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.305805    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-tmp-dir\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.307710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.307679    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-dshm\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.307909 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.307894    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb89111d-7a71-4de1-994a-7470dc21d733-tls-certs\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.334021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.333970    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb26w\" (UniqueName: \"kubernetes.io/projected/eb89111d-7a71-4de1-994a-7470dc21d733-kube-api-access-xb26w\") pod \"auth-test-model-75443e-kserve-6777b67757-85xmh\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") " pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.438381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.438305    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:32.711408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.711331    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"]
Oct 30 14:33:32.711968 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.711953    4530 cpu_manager.go:414] "containerMap: removing container" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy"
Oct 30 14:33:32.712010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.711970    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy"
Oct 30 14:33:32.712010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.711981    4530 cpu_manager.go:414] "containerMap: removing container" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy"
Oct 30 14:33:32.712010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.711986    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06147d62-09b3-4e0e-bb3a-a8fad254a8dd" containerName="istio-proxy"
Oct 30 14:33:32.712115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.712106    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83aae0b-4976-4a91-937c-c1e3dda7f151" containerName="istio-proxy"
Oct 30 14:33:32.716533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.716513    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:32.738314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.738288    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6e91df-serving-cert\""
Oct 30 14:33:32.738577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.738555    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"]
Oct 30 14:33:32.757374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.757342    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"]
Oct 30 14:33:32.762082 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:33:32.762050    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb89111d_7a71_4de1_994a_7470dc21d733.slice/crio-ad09283ca6ee5d2b99bf32d3e476f997d63a0d0e19d543b2fe0045a8a9479458 WatchSource:0}: Error finding container ad09283ca6ee5d2b99bf32d3e476f997d63a0d0e19d543b2fe0045a8a9479458: Status 404 returned error can't find the container with id ad09283ca6ee5d2b99bf32d3e476f997d63a0d0e19d543b2fe0045a8a9479458
Oct 30 14:33:32.812093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.812065    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:32.812234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.812166    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z62p2\" (UniqueName: \"kubernetes.io/projected/940bb1b7-7692-4676-8b62-5f283e2b0811-kube-api-access-z62p2\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:32.812297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.812252    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940bb1b7-7692-4676-8b62-5f283e2b0811-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:32.812354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.812308    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:32.913137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.913103    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940bb1b7-7692-4676-8b62-5f283e2b0811-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:32.913340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.913155    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:32.913340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.913271    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:32.913466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.913361    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z62p2\" (UniqueName: \"kubernetes.io/projected/940bb1b7-7692-4676-8b62-5f283e2b0811-kube-api-access-z62p2\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:32.913868 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:33:32.913845    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-serving-cert: secret "maas-api-e2e-shared-b-w2-6e91df-serving-cert" not found
Oct 30 14:33:32.913969 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:33:32.913917    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-maas-api-tls podName:940bb1b7-7692-4676-8b62-5f283e2b0811 nodeName:}" failed. No retries permitted until 2026-10-30 14:33:33.413895482 +0000 UTC m=+2678.445794415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-maas-api-tls") pod "maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl" (UID: "940bb1b7-7692-4676-8b62-5f283e2b0811") : secret "maas-api-e2e-shared-b-w2-6e91df-serving-cert" not found
Oct 30 14:33:32.914042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.913958    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940bb1b7-7692-4676-8b62-5f283e2b0811-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:32.919584 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.919048    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:32.949439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.949233    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z62p2\" (UniqueName: \"kubernetes.io/projected/940bb1b7-7692-4676-8b62-5f283e2b0811-kube-api-access-z62p2\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:32.958292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.958259    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh" event={"ID":"eb89111d-7a71-4de1-994a-7470dc21d733","Type":"ContainerStarted","Data":"46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e"}
Oct 30 14:33:32.958425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:32.958302    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh" event={"ID":"eb89111d-7a71-4de1-994a-7470dc21d733","Type":"ContainerStarted","Data":"ad09283ca6ee5d2b99bf32d3e476f997d63a0d0e19d543b2fe0045a8a9479458"}
Oct 30 14:33:33.018719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.018640    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"]
Oct 30 14:33:33.024683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.024655    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.042889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.042834    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh" podStartSLOduration=1.042816574 podStartE2EDuration="1.042816574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:33.02565614 +0000 UTC m=+2678.057555087" watchObservedRunningTime="2026-10-30 14:33:33.042816574 +0000 UTC m=+2678.074715514"
Oct 30 14:33:33.044488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.044463    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-9e3aa0-openshift-default-dockercfg-n4l2x\""
Oct 30 14:33:33.044802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.044780    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"]
Oct 30 14:33:33.130578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.128494    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-certs\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.130578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.128591    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w4h9\" (UniqueName: \"kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-kube-api-access-8w4h9\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.130578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.128634    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-data\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.130578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.128729    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istiod-ca-cert\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.130578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.128832    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-credential-socket\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.130578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.128893    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-podinfo\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.130578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.128938    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-socket\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.130578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.129034    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-token\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.130578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.129114    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-envoy\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.230565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.230331    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-podinfo\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.230565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.230402    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-socket\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.230827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.230600    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-token\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.230827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.230709    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-envoy\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.230827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.230797    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-certs\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.231011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.230832    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w4h9\" (UniqueName: \"kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-kube-api-access-8w4h9\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.231011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.230881    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-data\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.231011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.230914    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istiod-ca-cert\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.231185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.231036    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-credential-socket\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.234559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.231619    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-credential-socket\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.234559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.232910    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-certs\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.234559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.233352    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-socket\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.234559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.233348    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-data\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.234559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.233811    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istiod-ca-cert\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.236582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.235158    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-podinfo\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.236582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.235321    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-envoy\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.266522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.266486    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-token\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.269152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.268153    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w4h9\" (UniqueName: \"kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-kube-api-access-8w4h9\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.342966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.342930    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:33.437219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.436523    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:33.438705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.438614    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:33.448852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.444244    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:33.631282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.631249    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:33.663827 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:33:33.663729    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc59ec5_9f0f_481b_8d34_f5e22619f51e.slice/crio-4d9f0ee031c16eb492155aacb481ac4f85b8ad4d02fcd2f39ad77992737731f3 WatchSource:0}: Error finding container 4d9f0ee031c16eb492155aacb481ac4f85b8ad4d02fcd2f39ad77992737731f3: Status 404 returned error can't find the container with id 4d9f0ee031c16eb492155aacb481ac4f85b8ad4d02fcd2f39ad77992737731f3
Oct 30 14:33:33.665639 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.665618    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"]
Oct 30 14:33:33.667484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.667449    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:33:33.667605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.667513    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:33:33.667605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.667557    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:33:33.941466 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:33:33.941415    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940bb1b7_7692_4676_8b62_5f283e2b0811.slice/crio-88f78076841fefb49dcc1122bb8d128009fc7fbfc8f4910396a250a062e4afcf WatchSource:0}: Error finding container 88f78076841fefb49dcc1122bb8d128009fc7fbfc8f4910396a250a062e4afcf: Status 404 returned error can't find the container with id 88f78076841fefb49dcc1122bb8d128009fc7fbfc8f4910396a250a062e4afcf
Oct 30 14:33:33.941622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.941567    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"]
Oct 30 14:33:33.967231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.967197    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" event={"ID":"6bc59ec5-9f0f-481b-8d34-f5e22619f51e","Type":"ContainerStarted","Data":"16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516"}
Oct 30 14:33:33.967346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.967253    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" event={"ID":"6bc59ec5-9f0f-481b-8d34-f5e22619f51e","Type":"ContainerStarted","Data":"4d9f0ee031c16eb492155aacb481ac4f85b8ad4d02fcd2f39ad77992737731f3"}
Oct 30 14:33:33.969385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:33.969357    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl" event={"ID":"940bb1b7-7692-4676-8b62-5f283e2b0811","Type":"ContainerStarted","Data":"88f78076841fefb49dcc1122bb8d128009fc7fbfc8f4910396a250a062e4afcf"}
Oct 30 14:33:34.021146 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:34.021104    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" podStartSLOduration=2.021091335 podStartE2EDuration="2.021091335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:34.020162588 +0000 UTC m=+2679.052061539" watchObservedRunningTime="2026-10-30 14:33:34.021091335 +0000 UTC m=+2679.052990273"
Oct 30 14:33:34.343500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:34.343474    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:34.349533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:34.349508    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:34.978562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:34.978510    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl" event={"ID":"940bb1b7-7692-4676-8b62-5f283e2b0811","Type":"ContainerStarted","Data":"0830240c7a31e65ab8063cd5bddd332117ed31679f15af224f79625ff3c034be"}
Oct 30 14:33:34.978769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:34.978612    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:34.979126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:34.979101    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:34.980128 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:34.980109    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:33:35.027400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:35.027356    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl" podStartSLOduration=2.724940367 podStartE2EDuration="3.027342607s" totalImagesPullingTime="302.40224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:35.025167546 +0000 UTC m=+2680.057066485" watchObservedRunningTime="2026-10-30 14:33:35.027342607 +0000 UTC m=+2680.059241546"
Oct 30 14:33:38.009102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.009068    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"]
Oct 30 14:33:38.009640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.009341    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy" containerID="cri-o://ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c" gracePeriod=30
Oct 30 14:33:38.268081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.268003    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"]
Oct 30 14:33:38.272806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.272784    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"
Oct 30 14:33:38.291224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.291201    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"]
Oct 30 14:33:38.291363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.291321    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-9e3aa0\""
Oct 30 14:33:38.291363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.291331    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-9e3aa0-dockercfg-dcm7n\""
Oct 30 14:33:38.416568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.416381    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5v7h\" (UniqueName: \"kubernetes.io/projected/8628b21b-fb85-4411-b29f-9b23ef9c05a8-kube-api-access-z5v7h\") pod \"payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f\" (UID: \"8628b21b-fb85-4411-b29f-9b23ef9c05a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"
Oct 30 14:33:38.416781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.416571    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8628b21b-fb85-4411-b29f-9b23ef9c05a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f\" (UID: \"8628b21b-fb85-4411-b29f-9b23ef9c05a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"
Oct 30 14:33:38.432613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.423653    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"]
Oct 30 14:33:38.434588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.434272    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"
Oct 30 14:33:38.438592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.435877    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"]
Oct 30 14:33:38.517729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.517698    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkprq\" (UniqueName: \"kubernetes.io/projected/06ef73ec-3562-49a8-a436-515403188436-kube-api-access-kkprq\") pod \"payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf\" (UID: \"06ef73ec-3562-49a8-a436-515403188436\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"
Oct 30 14:33:38.517908 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.517737    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8628b21b-fb85-4411-b29f-9b23ef9c05a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f\" (UID: \"8628b21b-fb85-4411-b29f-9b23ef9c05a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"
Oct 30 14:33:38.517955 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.517934    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5v7h\" (UniqueName: \"kubernetes.io/projected/8628b21b-fb85-4411-b29f-9b23ef9c05a8-kube-api-access-z5v7h\") pod \"payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f\" (UID: \"8628b21b-fb85-4411-b29f-9b23ef9c05a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"
Oct 30 14:33:38.518000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.517977    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ef73ec-3562-49a8-a436-515403188436-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf\" (UID: \"06ef73ec-3562-49a8-a436-515403188436\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"
Oct 30 14:33:38.518458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.518412    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8628b21b-fb85-4411-b29f-9b23ef9c05a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f\" (UID: \"8628b21b-fb85-4411-b29f-9b23ef9c05a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"
Oct 30 14:33:38.544851 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.544829    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5v7h\" (UniqueName: \"kubernetes.io/projected/8628b21b-fb85-4411-b29f-9b23ef9c05a8-kube-api-access-z5v7h\") pod \"payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f\" (UID: \"8628b21b-fb85-4411-b29f-9b23ef9c05a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"
Oct 30 14:33:38.588194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.588169    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"
Oct 30 14:33:38.619493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.619467    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkprq\" (UniqueName: \"kubernetes.io/projected/06ef73ec-3562-49a8-a436-515403188436-kube-api-access-kkprq\") pod \"payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf\" (UID: \"06ef73ec-3562-49a8-a436-515403188436\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"
Oct 30 14:33:38.619670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.619651    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ef73ec-3562-49a8-a436-515403188436-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf\" (UID: \"06ef73ec-3562-49a8-a436-515403188436\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"
Oct 30 14:33:38.620711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.620684    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ef73ec-3562-49a8-a436-515403188436-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf\" (UID: \"06ef73ec-3562-49a8-a436-515403188436\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"
Oct 30 14:33:38.647362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.647336    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkprq\" (UniqueName: \"kubernetes.io/projected/06ef73ec-3562-49a8-a436-515403188436-kube-api-access-kkprq\") pod \"payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf\" (UID: \"06ef73ec-3562-49a8-a436-515403188436\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"
Oct 30 14:33:38.772054 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.771974    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"
Oct 30 14:33:38.900959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:38.900931    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"]
Oct 30 14:33:38.901165 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:33:38.901130    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8628b21b_fb85_4411_b29f_9b23ef9c05a8.slice/crio-62edf44f70fd2edfd981387a42a2a3a0b8a9e22f0b39c695c6dfd82a2c14b0ef WatchSource:0}: Error finding container 62edf44f70fd2edfd981387a42a2a3a0b8a9e22f0b39c695c6dfd82a2c14b0ef: Status 404 returned error can't find the container with id 62edf44f70fd2edfd981387a42a2a3a0b8a9e22f0b39c695c6dfd82a2c14b0ef
Oct 30 14:33:39.007607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:39.007571    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f" event={"ID":"8628b21b-fb85-4411-b29f-9b23ef9c05a8","Type":"ContainerStarted","Data":"6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833"}
Oct 30 14:33:39.007719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:39.007673    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f" event={"ID":"8628b21b-fb85-4411-b29f-9b23ef9c05a8","Type":"ContainerStarted","Data":"62edf44f70fd2edfd981387a42a2a3a0b8a9e22f0b39c695c6dfd82a2c14b0ef"}
Oct 30 14:33:39.007834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:39.007814    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"
Oct 30 14:33:39.057651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:39.057605    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f" podStartSLOduration=1.057592236 podStartE2EDuration="1.057592236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:39.055653824 +0000 UTC m=+2684.087552763" watchObservedRunningTime="2026-10-30 14:33:39.057592236 +0000 UTC m=+2684.089491175"
Oct 30 14:33:39.083659 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:33:39.083630    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ef73ec_3562_49a8_a436_515403188436.slice/crio-e5b7850f47bbb2d3d3c978bbc8f9eff13c1085754ee959623848fc6b3fa2f0f7 WatchSource:0}: Error finding container e5b7850f47bbb2d3d3c978bbc8f9eff13c1085754ee959623848fc6b3fa2f0f7: Status 404 returned error can't find the container with id e5b7850f47bbb2d3d3c978bbc8f9eff13c1085754ee959623848fc6b3fa2f0f7
Oct 30 14:33:39.084990 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:39.084964    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"]
Oct 30 14:33:40.021843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:40.021801    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf" event={"ID":"06ef73ec-3562-49a8-a436-515403188436","Type":"ContainerStarted","Data":"3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf"}
Oct 30 14:33:40.022045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:40.021873    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"
Oct 30 14:33:40.022045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:40.021898    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf" event={"ID":"06ef73ec-3562-49a8-a436-515403188436","Type":"ContainerStarted","Data":"e5b7850f47bbb2d3d3c978bbc8f9eff13c1085754ee959623848fc6b3fa2f0f7"}
Oct 30 14:33:40.072469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:40.072419    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf" podStartSLOduration=2.072405664 podStartE2EDuration="2.072405664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:40.070696219 +0000 UTC m=+2685.102595157" watchObservedRunningTime="2026-10-30 14:33:40.072405664 +0000 UTC m=+2685.104304603"
Oct 30 14:33:40.990807 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:40.990781    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:33:41.368665 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:33:41.368624    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:33:42.439433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:42.439384    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:42.454258 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:42.454224    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:43.073235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.073209    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:33:43.284159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.284103    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:33:43.389062 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389029    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad83b9c7-59fd-40da-9726-98f8366e3163-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad83b9c7-59fd-40da-9726-98f8366e3163-istiod-ca-cert\") pod \"ad83b9c7-59fd-40da-9726-98f8366e3163\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") "
Oct 30 14:33:43.389259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389124    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-certs\") pod \"ad83b9c7-59fd-40da-9726-98f8366e3163\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") "
Oct 30 14:33:43.389259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389158    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-podinfo\") pod \"ad83b9c7-59fd-40da-9726-98f8366e3163\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") "
Oct 30 14:33:43.389259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389249    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-socket\") pod \"ad83b9c7-59fd-40da-9726-98f8366e3163\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") "
Oct 30 14:33:43.389442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389277    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-credential-socket\") pod \"ad83b9c7-59fd-40da-9726-98f8366e3163\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") "
Oct 30 14:33:43.389442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389297    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-data\") pod \"ad83b9c7-59fd-40da-9726-98f8366e3163\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") "
Oct 30 14:33:43.389442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389329    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-token\" (UniqueName: \"kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-token\") pod \"ad83b9c7-59fd-40da-9726-98f8366e3163\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") "
Oct 30 14:33:43.389442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389376    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-kube-api-access-88hb4\" (UniqueName: \"kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-kube-api-access-88hb4\") pod \"ad83b9c7-59fd-40da-9726-98f8366e3163\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") "
Oct 30 14:33:43.389688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389442    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad83b9c7-59fd-40da-9726-98f8366e3163-istiod-ca-cert" pod "ad83b9c7-59fd-40da-9726-98f8366e3163" (UID: "ad83b9c7-59fd-40da-9726-98f8366e3163"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:33:43.389688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389509    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-envoy\") pod \"ad83b9c7-59fd-40da-9726-98f8366e3163\" (UID: \"ad83b9c7-59fd-40da-9726-98f8366e3163\") "
Oct 30 14:33:43.389688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389590    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-socket" pod "ad83b9c7-59fd-40da-9726-98f8366e3163" (UID: "ad83b9c7-59fd-40da-9726-98f8366e3163"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:43.389688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389444    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-certs" pod "ad83b9c7-59fd-40da-9726-98f8366e3163" (UID: "ad83b9c7-59fd-40da-9726-98f8366e3163"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:43.389915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.389757    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-credential-socket" pod "ad83b9c7-59fd-40da-9726-98f8366e3163" (UID: "ad83b9c7-59fd-40da-9726-98f8366e3163"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:43.390100 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.390081    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad83b9c7-59fd-40da-9726-98f8366e3163-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:43.390178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.390106    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:43.390178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.390119    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:43.390178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.390136    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:43.390361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.390340    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-data" pod "ad83b9c7-59fd-40da-9726-98f8366e3163" (UID: "ad83b9c7-59fd-40da-9726-98f8366e3163"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:43.391769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.391738    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-podinfo" pod "ad83b9c7-59fd-40da-9726-98f8366e3163" (UID: "ad83b9c7-59fd-40da-9726-98f8366e3163"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:33:43.391769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.391760    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-token" pod "ad83b9c7-59fd-40da-9726-98f8366e3163" (UID: "ad83b9c7-59fd-40da-9726-98f8366e3163"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:33:43.392074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.392057    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-envoy" pod "ad83b9c7-59fd-40da-9726-98f8366e3163" (UID: "ad83b9c7-59fd-40da-9726-98f8366e3163"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:43.392139 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.392123    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-kube-api-access-88hb4" pod "ad83b9c7-59fd-40da-9726-98f8366e3163" (UID: "ad83b9c7-59fd-40da-9726-98f8366e3163"). InnerVolumeSpecName "kube-api-access-88hb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:33:43.491824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.491790    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:43.491824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.491821    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:43.491824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.491831    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:43.492316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.491840    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:43.492316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:43.491849    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88hb4\" (UniqueName: \"kubernetes.io/projected/ad83b9c7-59fd-40da-9726-98f8366e3163-kube-api-access-88hb4\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:44.070776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:44.070735    4530 generic.go:356] "Generic (PLEG): container finished" podID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerID="ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c" exitCode=0
Oct 30 14:33:44.070956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:44.070795    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"
Oct 30 14:33:44.070956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:44.070793    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" event={"ID":"ad83b9c7-59fd-40da-9726-98f8366e3163","Type":"ContainerDied","Data":"ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c"}
Oct 30 14:33:44.070956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:44.070943    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4" event={"ID":"ad83b9c7-59fd-40da-9726-98f8366e3163","Type":"ContainerDied","Data":"7aa7c04779dea7f48b407f9122760f768e339fbb186e6cc55615b0f102ab9c25"}
Oct 30 14:33:44.071157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:44.070964    4530 scope.go:122] "RemoveContainer" containerID="ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c"
Oct 30 14:33:44.083039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:44.083014    4530 scope.go:122] "RemoveContainer" containerID="ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c"
Oct 30 14:33:44.083276 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:33:44.083252    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c\": container with ID starting with ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c not found: ID does not exist" containerID="ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c"
Oct 30 14:33:44.083371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:44.083282    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c"} err="failed to get container status \"ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c\": rpc error: code = NotFound desc = could not find container \"ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c\": container with ID starting with ef6685d0227e24ebf9de878bf9c78e2815b8cd53be7eef99b46c53dcd0fc105c not found: ID does not exist"
Oct 30 14:33:44.148725 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:44.148694    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"]
Oct 30 14:33:44.165788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:44.165760    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d2gh4"]
Oct 30 14:33:45.114947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:45.114915    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" path="/var/lib/kubelet/pods/ad83b9c7-59fd-40da-9726-98f8366e3163/volumes"
Oct 30 14:33:45.661121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:45.661084    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb" podUID="f0e85c33-012d-4136-abf2-029fb2600db2" containerName="main" containerID="cri-o://1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232" gracePeriod=45
Oct 30 14:33:45.931289 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:45.931225    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:33:46.088077 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.088044    4530 generic.go:356] "Generic (PLEG): container finished" podID="f0e85c33-012d-4136-abf2-029fb2600db2" containerID="1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232" exitCode=0
Oct 30 14:33:46.088252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.088087    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"
Oct 30 14:33:46.088252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.088134    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb" event={"ID":"f0e85c33-012d-4136-abf2-029fb2600db2","Type":"ContainerDied","Data":"1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232"}
Oct 30 14:33:46.088252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.088180    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb" event={"ID":"f0e85c33-012d-4136-abf2-029fb2600db2","Type":"ContainerDied","Data":"1129ce7007def886edf1969a478259baa364dd62ea50dacb2d83037288eb6b9c"}
Oct 30 14:33:46.088252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.088201    4530 scope.go:122] "RemoveContainer" containerID="1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232"
Oct 30 14:33:46.098321 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.098304    4530 scope.go:122] "RemoveContainer" containerID="1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232"
Oct 30 14:33:46.098593 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:33:46.098573    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232\": container with ID starting with 1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232 not found: ID does not exist" containerID="1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232"
Oct 30 14:33:46.098692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.098601    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232"} err="failed to get container status \"1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232\": rpc error: code = NotFound desc = could not find container \"1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232\": container with ID starting with 1f99603c15a6e10999ea8eba4cf4c7458919a323c41a54f0279465e3b5c2a232 not found: ID does not exist"
Oct 30 14:33:46.120400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.120371    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0e85c33-012d-4136-abf2-029fb2600db2-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e85c33-012d-4136-abf2-029fb2600db2-tls-certs\") pod \"f0e85c33-012d-4136-abf2-029fb2600db2\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") "
Oct 30 14:33:46.120836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.120419    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e85c33-012d-4136-abf2-029fb2600db2-kube-api-access-hsg65\" (UniqueName: \"kubernetes.io/projected/f0e85c33-012d-4136-abf2-029fb2600db2-kube-api-access-hsg65\") pod \"f0e85c33-012d-4136-abf2-029fb2600db2\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") "
Oct 30 14:33:46.120836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.120581    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-home\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-home\") pod \"f0e85c33-012d-4136-abf2-029fb2600db2\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") "
Oct 30 14:33:46.120836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.120646    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-model-cache\") pod \"f0e85c33-012d-4136-abf2-029fb2600db2\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") "
Oct 30 14:33:46.120836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.120679    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-dshm\") pod \"f0e85c33-012d-4136-abf2-029fb2600db2\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") "
Oct 30 14:33:46.120836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.120726    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-tmp-dir\") pod \"f0e85c33-012d-4136-abf2-029fb2600db2\" (UID: \"f0e85c33-012d-4136-abf2-029fb2600db2\") "
Oct 30 14:33:46.120836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.120820    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-home" pod "f0e85c33-012d-4136-abf2-029fb2600db2" (UID: "f0e85c33-012d-4136-abf2-029fb2600db2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:46.121116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.120952    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-model-cache" pod "f0e85c33-012d-4136-abf2-029fb2600db2" (UID: "f0e85c33-012d-4136-abf2-029fb2600db2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:46.121116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.121097    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-tmp-dir" pod "f0e85c33-012d-4136-abf2-029fb2600db2" (UID: "f0e85c33-012d-4136-abf2-029fb2600db2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:46.121245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.121229    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:46.121284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.121251    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:46.121284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.121264    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:46.122781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.122752    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e85c33-012d-4136-abf2-029fb2600db2-kube-api-access-hsg65" pod "f0e85c33-012d-4136-abf2-029fb2600db2" (UID: "f0e85c33-012d-4136-abf2-029fb2600db2"). InnerVolumeSpecName "kube-api-access-hsg65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:33:46.122781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.122757    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-dshm" pod "f0e85c33-012d-4136-abf2-029fb2600db2" (UID: "f0e85c33-012d-4136-abf2-029fb2600db2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:33:46.122912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.122776    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e85c33-012d-4136-abf2-029fb2600db2-tls-certs" pod "f0e85c33-012d-4136-abf2-029fb2600db2" (UID: "f0e85c33-012d-4136-abf2-029fb2600db2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:33:46.222617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.222537    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e85c33-012d-4136-abf2-029fb2600db2-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:46.222617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.222574    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsg65\" (UniqueName: \"kubernetes.io/projected/f0e85c33-012d-4136-abf2-029fb2600db2-kube-api-access-hsg65\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:46.222617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.222583    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e85c33-012d-4136-abf2-029fb2600db2-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:33:46.441484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.441452    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"]
Oct 30 14:33:46.455310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:46.455285    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1d3696/test-model-1d3696-kserve-7fb699d56f-jmqjb"]
Oct 30 14:33:47.116184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:47.116153    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e85c33-012d-4136-abf2-029fb2600db2" path="/var/lib/kubelet/pods/f0e85c33-012d-4136-abf2-029fb2600db2/volumes"
Oct 30 14:33:50.024697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:50.024613    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"
Oct 30 14:33:51.032858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:51.032826    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"
Oct 30 14:33:54.520757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:54.520723    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:33:55.933527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:55.933495    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:33:55.933987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:55.933614    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:33:55.939956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:55.939935    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:33:55.940088 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:55.939986    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:33:55.941102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:55.941085    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:33:55.941246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:33:55.941225    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:34:00.159575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.159385    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"]
Oct 30 14:34:00.160611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.160585    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f0e85c33-012d-4136-abf2-029fb2600db2" containerName="main"
Oct 30 14:34:00.160611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.160609    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e85c33-012d-4136-abf2-029fb2600db2" containerName="main"
Oct 30 14:34:00.160795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.160669    4530 cpu_manager.go:414] "containerMap: removing container" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy"
Oct 30 14:34:00.160795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.160677    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy"
Oct 30 14:34:00.160916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.160868    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e85c33-012d-4136-abf2-029fb2600db2" containerName="main"
Oct 30 14:34:00.160916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.160899    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad83b9c7-59fd-40da-9726-98f8366e3163" containerName="istio-proxy"
Oct 30 14:34:00.165057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.165034    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.214962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.214926    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"]
Oct 30 14:34:00.221129 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.221103    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5704d6\"/\"kube-root-ca.crt\""
Oct 30 14:34:00.221316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.221198    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5704d6\"/\"auth-test-model-5704d6-kserve-self-signed-certs\""
Oct 30 14:34:00.221431 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.221242    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5704d6\"/\"openshift-service-ca.crt\""
Oct 30 14:34:00.221568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.221286    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5704d6\"/\"default-dockercfg-87hr6\""
Oct 30 14:34:00.343694 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.343662    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-model-cache\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.343868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.343727    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tmp-dir\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.343868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.343814    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-dshm\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.343960 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.343922    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-home\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.343997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.343985    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq4sr\" (UniqueName: \"kubernetes.io/projected/23ff4fb9-73ff-4d09-a9a4-125a23a52995-kube-api-access-pq4sr\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.344056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.344019    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tls-certs\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.463893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.457852    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq4sr\" (UniqueName: \"kubernetes.io/projected/23ff4fb9-73ff-4d09-a9a4-125a23a52995-kube-api-access-pq4sr\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.463893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.457931    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tls-certs\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.463893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.458055    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-model-cache\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.463893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.458108    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tmp-dir\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.463893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.458216    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-dshm\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.463893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.458311    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-home\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.463893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.458821    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-home\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.463893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.459856    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tmp-dir\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.463893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.460145    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-model-cache\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.468422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.468306    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-dshm\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.469180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.469154    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tls-certs\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.488950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.488918    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq4sr\" (UniqueName: \"kubernetes.io/projected/23ff4fb9-73ff-4d09-a9a4-125a23a52995-kube-api-access-pq4sr\") pod \"auth-test-model-5704d6-kserve-6cdd8c4949-bsknd\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") " pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:00.778259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:00.778176    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:01.127056 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:34:01.127015    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23ff4fb9_73ff_4d09_a9a4_125a23a52995.slice/crio-9cb6a82290c975ecbd5c152abd3181828bb32aad32d6e35815a78f131ce47ed1 WatchSource:0}: Error finding container 9cb6a82290c975ecbd5c152abd3181828bb32aad32d6e35815a78f131ce47ed1: Status 404 returned error can't find the container with id 9cb6a82290c975ecbd5c152abd3181828bb32aad32d6e35815a78f131ce47ed1
Oct 30 14:34:01.134303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:01.132822    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"]
Oct 30 14:34:01.232852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:01.232822    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd" event={"ID":"23ff4fb9-73ff-4d09-a9a4-125a23a52995","Type":"ContainerStarted","Data":"9cb6a82290c975ecbd5c152abd3181828bb32aad32d6e35815a78f131ce47ed1"}
Oct 30 14:34:02.241956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:02.241920    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd" event={"ID":"23ff4fb9-73ff-4d09-a9a4-125a23a52995","Type":"ContainerStarted","Data":"6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce"}
Oct 30 14:34:02.294555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:02.294494    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd" podStartSLOduration=2.294480147 podStartE2EDuration="2.294480147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:02.291067028 +0000 UTC m=+2707.322965967" watchObservedRunningTime="2026-10-30 14:34:02.294480147 +0000 UTC m=+2707.326379086"
Oct 30 14:34:02.779457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:02.779420    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:06.275366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:06.275284    4530 generic.go:356] "Generic (PLEG): container finished" podID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerID="6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4" exitCode=0
Oct 30 14:34:06.275780 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:06.275360    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" event={"ID":"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3","Type":"ContainerDied","Data":"6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4"}
Oct 30 14:34:06.275828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:06.275803    4530 scope.go:122] "RemoveContainer" containerID="6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4"
Oct 30 14:34:06.277529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:06.277492    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:34:06.277625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:06.277584    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:34:06.277667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:06.277624    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:34:07.309736 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:07.309701    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" event={"ID":"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3","Type":"ContainerStarted","Data":"ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c"}
Oct 30 14:34:08.202277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:08.202243    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:34:08.204677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:08.204642    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 14:34:08.204798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:08.204740    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 14:34:09.193573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:09.191663    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 14:34:09.193573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:09.191732    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 14:34:10.197566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:10.194955    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 14:34:10.197566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:10.195083    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 14:34:10.500369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:10.500280    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:34:10.779330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:10.779248    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:10.819698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:10.819664    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:11.205567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:11.203928    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 14:34:11.205567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:11.204005    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 14:34:11.349914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:11.349888    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:34:11.368006 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:34:11.367974    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:34:12.182719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.182669    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 14:34:12.182933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.182736    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 14:34:12.780358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.780318    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"]
Oct 30 14:34:12.784967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.784944    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.818680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.818653    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-0efe43\"/\"openshift-service-ca.crt\""
Oct 30 14:34:12.818971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.818943    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-0efe43\"/\"auth-test-model-0efe43-kserve-self-signed-certs\""
Oct 30 14:34:12.819093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.818978    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-0efe43\"/\"default-dockercfg-sb4ng\""
Oct 30 14:34:12.819093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.819046    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-0efe43\"/\"kube-root-ca.crt\""
Oct 30 14:34:12.825512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.825488    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"]
Oct 30 14:34:12.841310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.841282    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3db87e42-382c-4022-8a55-574b4edc1344-tls-certs\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.841455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.841339    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-model-cache\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.841520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.841507    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtpmz\" (UniqueName: \"kubernetes.io/projected/3db87e42-382c-4022-8a55-574b4edc1344-kube-api-access-wtpmz\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.841649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.841627    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-home\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.841710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.841690    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-tmp-dir\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.841772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.841741    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-dshm\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.943150 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.943119    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3db87e42-382c-4022-8a55-574b4edc1344-tls-certs\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.943300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.943162    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-model-cache\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.943300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.943225    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtpmz\" (UniqueName: \"kubernetes.io/projected/3db87e42-382c-4022-8a55-574b4edc1344-kube-api-access-wtpmz\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.943300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.943276    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-home\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.943468 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.943308    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-tmp-dir\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.943468 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.943339    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-dshm\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.943746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.943720    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-model-cache\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.943819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.943745    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-home\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.943819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.943785    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-tmp-dir\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.945533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.945514    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-dshm\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.945711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.945694    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3db87e42-382c-4022-8a55-574b4edc1344-tls-certs\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:12.971636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:12.971613    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtpmz\" (UniqueName: \"kubernetes.io/projected/3db87e42-382c-4022-8a55-574b4edc1344-kube-api-access-wtpmz\") pod \"auth-test-model-0efe43-kserve-64b47fdcc-lch7p\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:13.100235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:13.100197    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:13.187385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:13.187356    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:34:13.188777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:13.188737    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:34:13.418230 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:34:13.418194    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3db87e42_382c_4022_8a55_574b4edc1344.slice/crio-4ade57ced1e40ef57a335404fd68588b34b8d787c803648a6513e345c568c901 WatchSource:0}: Error finding container 4ade57ced1e40ef57a335404fd68588b34b8d787c803648a6513e345c568c901: Status 404 returned error can't find the container with id 4ade57ced1e40ef57a335404fd68588b34b8d787c803648a6513e345c568c901
Oct 30 14:34:13.420298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:13.420271    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"]
Oct 30 14:34:14.371053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:14.371017    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p" event={"ID":"3db87e42-382c-4022-8a55-574b4edc1344","Type":"ContainerStarted","Data":"c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191"}
Oct 30 14:34:14.371053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:14.371059    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p" event={"ID":"3db87e42-382c-4022-8a55-574b4edc1344","Type":"ContainerStarted","Data":"4ade57ced1e40ef57a335404fd68588b34b8d787c803648a6513e345c568c901"}
Oct 30 14:34:14.420027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:14.419981    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p" podStartSLOduration=2.419967354 podStartE2EDuration="2.419967354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:14.417732638 +0000 UTC m=+2719.449631590" watchObservedRunningTime="2026-10-30 14:34:14.419967354 +0000 UTC m=+2719.451866307"
Oct 30 14:34:15.101607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.101578    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:15.572298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.572268    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"]
Oct 30 14:34:15.575520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.575503    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.598165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.598138    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"]
Oct 30 14:34:15.605728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.605703    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-9e3aa0-serving-cert\""
Oct 30 14:34:15.625432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.625399    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.625632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.625530    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktskp\" (UniqueName: \"kubernetes.io/projected/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-kube-api-access-ktskp\") pod \"maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.625711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.625661    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.625946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.625923    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.727593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.727566    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.727738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.727602    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.727738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.727646    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktskp\" (UniqueName: \"kubernetes.io/projected/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-kube-api-access-ktskp\") pod \"maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.727738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.727682    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.728176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.728148    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.730090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.730068    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.730223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.730199    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.779886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.779848    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktskp\" (UniqueName: \"kubernetes.io/projected/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-kube-api-access-ktskp\") pod \"maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:15.888778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:15.888705    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:16.193158 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:34:16.193134    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab4de72_e77d_46aa_afb2_f6cde66ef7b9.slice/crio-cfc12daa40f405bbc643faa86902df25d33e1c1632827d34c7a58080c7411f66 WatchSource:0}: Error finding container cfc12daa40f405bbc643faa86902df25d33e1c1632827d34c7a58080c7411f66: Status 404 returned error can't find the container with id cfc12daa40f405bbc643faa86902df25d33e1c1632827d34c7a58080c7411f66
Oct 30 14:34:16.194536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:16.194512    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"]
Oct 30 14:34:16.386083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:16.386047    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq" event={"ID":"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9","Type":"ContainerStarted","Data":"cfc12daa40f405bbc643faa86902df25d33e1c1632827d34c7a58080c7411f66"}
Oct 30 14:34:17.394396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:17.394362    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq" event={"ID":"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9","Type":"ContainerStarted","Data":"02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00"}
Oct 30 14:34:17.394835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:17.394493    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:17.445569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:17.445507    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq" podStartSLOduration=2.101245427 podStartE2EDuration="2.445494544s" totalImagesPullingTime="344.249117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:17.441868487 +0000 UTC m=+2722.473767439" watchObservedRunningTime="2026-10-30 14:34:17.445494544 +0000 UTC m=+2722.477393482"
Oct 30 14:34:21.142055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.142015    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"]
Oct 30 14:34:21.142604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.142256    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq" podUID="7aeffd3a-6f61-4cca-8a77-0b2a337639f2" containerName="maas-api" containerID="cri-o://5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c" gracePeriod=30
Oct 30 14:34:21.457552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.457207    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:34:21.460699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.460448    4530 generic.go:356] "Generic (PLEG): container finished" podID="7aeffd3a-6f61-4cca-8a77-0b2a337639f2" containerID="5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c" exitCode=0
Oct 30 14:34:21.460699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.460512    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq" event={"ID":"7aeffd3a-6f61-4cca-8a77-0b2a337639f2","Type":"ContainerDied","Data":"5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c"}
Oct 30 14:34:21.460699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.460585    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq" event={"ID":"7aeffd3a-6f61-4cca-8a77-0b2a337639f2","Type":"ContainerDied","Data":"2f3894fe316a7f9344e473f945ffe3086103e17c04397cff76c92d09ccf1b791"}
Oct 30 14:34:21.460699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.460610    4530 scope.go:122] "RemoveContainer" containerID="5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c"
Oct 30 14:34:21.460699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.460630    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"
Oct 30 14:34:21.472814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.472794    4530 scope.go:122] "RemoveContainer" containerID="5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c"
Oct 30 14:34:21.473122 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:34:21.473098    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c\": container with ID starting with 5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c not found: ID does not exist" containerID="5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c"
Oct 30 14:34:21.473226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.473132    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c"} err="failed to get container status \"5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c\": rpc error: code = NotFound desc = could not find container \"5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c\": container with ID starting with 5214e7245db95d0a0df3631c61fd22d256bfe179fe637a86f7e8d5726439ea8c not found: ID does not exist"
Oct 30 14:34:21.614340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.614307    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-openshift-service-ca\") pod \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") "
Oct 30 14:34:21.614501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.614397    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-kube-api-access-mngfp\" (UniqueName: \"kubernetes.io/projected/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-kube-api-access-mngfp\") pod \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") "
Oct 30 14:34:21.614501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.614427    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-maas-api-tls\") pod \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") "
Oct 30 14:34:21.614662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.614594    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-metrics-certs\") pod \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\" (UID: \"7aeffd3a-6f61-4cca-8a77-0b2a337639f2\") "
Oct 30 14:34:21.614768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.614740    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-openshift-service-ca" pod "7aeffd3a-6f61-4cca-8a77-0b2a337639f2" (UID: "7aeffd3a-6f61-4cca-8a77-0b2a337639f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:34:21.615075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.615055    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:34:21.616613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.616591    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-metrics-certs" pod "7aeffd3a-6f61-4cca-8a77-0b2a337639f2" (UID: "7aeffd3a-6f61-4cca-8a77-0b2a337639f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:34:21.616698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.616619    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-maas-api-tls" pod "7aeffd3a-6f61-4cca-8a77-0b2a337639f2" (UID: "7aeffd3a-6f61-4cca-8a77-0b2a337639f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:34:21.616768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.616751    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-kube-api-access-mngfp" pod "7aeffd3a-6f61-4cca-8a77-0b2a337639f2" (UID: "7aeffd3a-6f61-4cca-8a77-0b2a337639f2"). InnerVolumeSpecName "kube-api-access-mngfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:34:21.716493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.716456    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mngfp\" (UniqueName: \"kubernetes.io/projected/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-kube-api-access-mngfp\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:34:21.716493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.716490    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:34:21.716493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.716501    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aeffd3a-6f61-4cca-8a77-0b2a337639f2-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:34:21.818402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.818370    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"]
Oct 30 14:34:21.832306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:21.832276    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-019e5b-957dc9db7-4vxlq"]
Oct 30 14:34:23.101178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:23.101147    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:23.117681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:23.117650    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aeffd3a-6f61-4cca-8a77-0b2a337639f2" path="/var/lib/kubelet/pods/7aeffd3a-6f61-4cca-8a77-0b2a337639f2/volumes"
Oct 30 14:34:23.118075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:23.118051    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:23.403405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:23.403319    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq" podUID="9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.6:8443/health\": dial tcp 10.133.1.6:8443: connect: connection refused"
Oct 30 14:34:23.487291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:23.487260    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:34:27.504235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:27.504200    4530 generic.go:356] "Generic (PLEG): container finished" podID="9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" containerID="02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00" exitCode=1
Oct 30 14:34:27.504707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:27.504270    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq" event={"ID":"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9","Type":"ContainerDied","Data":"02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00"}
Oct 30 14:34:27.504707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:27.504645    4530 scope.go:122] "RemoveContainer" containerID="02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00"
Oct 30 14:34:28.513052 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:28.513011    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq" event={"ID":"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9","Type":"ContainerStarted","Data":"60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7"}
Oct 30 14:34:28.513513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:28.513223    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:34.524763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:34.524731    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:34:38.215617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:38.215517    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9600-f7a344"]
Oct 30 14:34:38.217386 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:38.217355    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7aeffd3a-6f61-4cca-8a77-0b2a337639f2" containerName="maas-api"
Oct 30 14:34:38.217386 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:38.217386    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aeffd3a-6f61-4cca-8a77-0b2a337639f2" containerName="maas-api"
Oct 30 14:34:38.217636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:38.217620    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aeffd3a-6f61-4cca-8a77-0b2a337639f2" containerName="maas-api"
Oct 30 14:34:38.221502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:38.221475    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9600-f7a344"]
Oct 30 14:34:38.221706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:38.221602    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9600-f7a344"
Oct 30 14:34:38.239994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:38.239972    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q95l7\""
Oct 30 14:34:38.365331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:38.365297    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvcfg\" (UniqueName: \"kubernetes.io/projected/c3bd77d9-6264-4625-be40-5815c7907002-kube-api-access-hvcfg\") pod \"test-curl-9600-f7a344\" (UID: \"c3bd77d9-6264-4625-be40-5815c7907002\") " pod="opendatahub/test-curl-9600-f7a344"
Oct 30 14:34:38.466453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:38.466373    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcfg\" (UniqueName: \"kubernetes.io/projected/c3bd77d9-6264-4625-be40-5815c7907002-kube-api-access-hvcfg\") pod \"test-curl-9600-f7a344\" (UID: \"c3bd77d9-6264-4625-be40-5815c7907002\") " pod="opendatahub/test-curl-9600-f7a344"
Oct 30 14:34:38.505214 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:38.505178    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvcfg\" (UniqueName: \"kubernetes.io/projected/c3bd77d9-6264-4625-be40-5815c7907002-kube-api-access-hvcfg\") pod \"test-curl-9600-f7a344\" (UID: \"c3bd77d9-6264-4625-be40-5815c7907002\") " pod="opendatahub/test-curl-9600-f7a344"
Oct 30 14:34:38.534409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:38.534382    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9600-f7a344"
Oct 30 14:34:38.864359 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:34:38.864326    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3bd77d9_6264_4625_be40_5815c7907002.slice/crio-3a5d6b8875e8c04ce46f91b7071d616cde3e5f922543a20aafb1200133aa321c WatchSource:0}: Error finding container 3a5d6b8875e8c04ce46f91b7071d616cde3e5f922543a20aafb1200133aa321c: Status 404 returned error can't find the container with id 3a5d6b8875e8c04ce46f91b7071d616cde3e5f922543a20aafb1200133aa321c
Oct 30 14:34:38.866091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:38.866063    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9600-f7a344"]
Oct 30 14:34:39.022503 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.022467    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"]
Oct 30 14:34:39.025697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.025674    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.048795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.048769    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"]
Oct 30 14:34:39.062255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.058183    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-396d70\"/\"kube-root-ca.crt\""
Oct 30 14:34:39.062255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.058223    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-396d70\"/\"openshift-service-ca.crt\""
Oct 30 14:34:39.063358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.063337    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-396d70\"/\"default-dockercfg-nb8cl\""
Oct 30 14:34:39.063358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.063335    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-396d70\"/\"sub-test-model-396d70-kserve-self-signed-certs\""
Oct 30 14:34:39.073887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.073865    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-tmp-dir\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.073996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.073895    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr5nb\" (UniqueName: \"kubernetes.io/projected/2a08e635-97e7-4723-94ae-a1c3b420fb52-kube-api-access-fr5nb\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.074042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.074017    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-model-cache\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.074089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.074053    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-dshm\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.074089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.074080    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-home\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.074324 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.074304    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a08e635-97e7-4723-94ae-a1c3b420fb52-tls-certs\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.175966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.175885    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-model-cache\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.175966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.175938    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-dshm\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.176219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.176045    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-home\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.176284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.176222    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a08e635-97e7-4723-94ae-a1c3b420fb52-tls-certs\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.176345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.176280    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-tmp-dir\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.176345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.176305    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-model-cache\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.176345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.176307    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr5nb\" (UniqueName: \"kubernetes.io/projected/2a08e635-97e7-4723-94ae-a1c3b420fb52-kube-api-access-fr5nb\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.176458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.176362    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-home\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.176661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.176632    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-tmp-dir\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.178259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.178231    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-dshm\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.178691 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.178672    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a08e635-97e7-4723-94ae-a1c3b420fb52-tls-certs\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.207185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.207151    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr5nb\" (UniqueName: \"kubernetes.io/projected/2a08e635-97e7-4723-94ae-a1c3b420fb52-kube-api-access-fr5nb\") pod \"sub-test-model-396d70-kserve-867db99ff9-t9dk4\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") " pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.340597 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.340531    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:39.645523 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.645488    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9600-f7a344" event={"ID":"c3bd77d9-6264-4625-be40-5815c7907002","Type":"ContainerStarted","Data":"52ae9c9c2d45efc2d3b58b792259818c90a7f478d16be31d306346e0b28b7383"}
Oct 30 14:34:39.645716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.645550    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9600-f7a344" event={"ID":"c3bd77d9-6264-4625-be40-5815c7907002","Type":"ContainerStarted","Data":"3a5d6b8875e8c04ce46f91b7071d616cde3e5f922543a20aafb1200133aa321c"}
Oct 30 14:34:39.671380 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:34:39.671350    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a08e635_97e7_4723_94ae_a1c3b420fb52.slice/crio-56a9d56c5566ddbc9271ef633be97be1bd9edffcacb7a5fe9b6f7e64f0445b27 WatchSource:0}: Error finding container 56a9d56c5566ddbc9271ef633be97be1bd9edffcacb7a5fe9b6f7e64f0445b27: Status 404 returned error can't find the container with id 56a9d56c5566ddbc9271ef633be97be1bd9edffcacb7a5fe9b6f7e64f0445b27
Oct 30 14:34:39.708249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.708220    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"]
Oct 30 14:34:39.711791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:39.711743    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9600-f7a344" podStartSLOduration=1.711729093 podStartE2EDuration="1.711729093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:39.695974388 +0000 UTC m=+2744.727873329" watchObservedRunningTime="2026-10-30 14:34:39.711729093 +0000 UTC m=+2744.743628075"
Oct 30 14:34:40.651185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:40.650199    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9600-f7a344"]
Oct 30 14:34:40.656576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:40.656416    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9600-f7a344"]
Oct 30 14:34:40.657551 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:40.657505    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4" event={"ID":"2a08e635-97e7-4723-94ae-a1c3b420fb52","Type":"ContainerStarted","Data":"12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed"}
Oct 30 14:34:40.657688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:40.657558    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4" event={"ID":"2a08e635-97e7-4723-94ae-a1c3b420fb52","Type":"ContainerStarted","Data":"56a9d56c5566ddbc9271ef633be97be1bd9edffcacb7a5fe9b6f7e64f0445b27"}
Oct 30 14:34:40.714101 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:40.714030    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4" podStartSLOduration=2.714007294 podStartE2EDuration="2.714007294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:40.710803104 +0000 UTC m=+2745.742702043" watchObservedRunningTime="2026-10-30 14:34:40.714007294 +0000 UTC m=+2745.745906235"
Oct 30 14:34:41.341158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:41.341126    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:41.367596 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:34:41.367564    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:34:41.667999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:41.667891    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9600-f7a344" podUID="c3bd77d9-6264-4625-be40-5815c7907002" containerName="test-curl-9600-f7a344" containerID="cri-o://52ae9c9c2d45efc2d3b58b792259818c90a7f478d16be31d306346e0b28b7383" gracePeriod=2
Oct 30 14:34:43.871837 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:43.871803    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9600-f7a344"
Oct 30 14:34:43.975896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:43.975821    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3bd77d9-6264-4625-be40-5815c7907002-kube-api-access-hvcfg\" (UniqueName: \"kubernetes.io/projected/c3bd77d9-6264-4625-be40-5815c7907002-kube-api-access-hvcfg\") pod \"c3bd77d9-6264-4625-be40-5815c7907002\" (UID: \"c3bd77d9-6264-4625-be40-5815c7907002\") "
Oct 30 14:34:43.977930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:43.977899    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3bd77d9-6264-4625-be40-5815c7907002-kube-api-access-hvcfg" pod "c3bd77d9-6264-4625-be40-5815c7907002" (UID: "c3bd77d9-6264-4625-be40-5815c7907002"). InnerVolumeSpecName "kube-api-access-hvcfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:34:44.077868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:44.077840    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvcfg\" (UniqueName: \"kubernetes.io/projected/c3bd77d9-6264-4625-be40-5815c7907002-kube-api-access-hvcfg\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:34:44.694457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:44.694409    4530 generic.go:356] "Generic (PLEG): container finished" podID="c3bd77d9-6264-4625-be40-5815c7907002" containerID="52ae9c9c2d45efc2d3b58b792259818c90a7f478d16be31d306346e0b28b7383" exitCode=137
Oct 30 14:34:44.694457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:44.694451    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9600-f7a344"
Oct 30 14:34:44.694768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:44.694504    4530 scope.go:122] "RemoveContainer" containerID="52ae9c9c2d45efc2d3b58b792259818c90a7f478d16be31d306346e0b28b7383"
Oct 30 14:34:44.707684 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:44.707663    4530 scope.go:122] "RemoveContainer" containerID="52ae9c9c2d45efc2d3b58b792259818c90a7f478d16be31d306346e0b28b7383"
Oct 30 14:34:44.707964 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:34:44.707943    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ae9c9c2d45efc2d3b58b792259818c90a7f478d16be31d306346e0b28b7383\": container with ID starting with 52ae9c9c2d45efc2d3b58b792259818c90a7f478d16be31d306346e0b28b7383 not found: ID does not exist" containerID="52ae9c9c2d45efc2d3b58b792259818c90a7f478d16be31d306346e0b28b7383"
Oct 30 14:34:44.708029 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:44.707976    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ae9c9c2d45efc2d3b58b792259818c90a7f478d16be31d306346e0b28b7383"} err="failed to get container status \"52ae9c9c2d45efc2d3b58b792259818c90a7f478d16be31d306346e0b28b7383\": rpc error: code = NotFound desc = could not find container \"52ae9c9c2d45efc2d3b58b792259818c90a7f478d16be31d306346e0b28b7383\": container with ID starting with 52ae9c9c2d45efc2d3b58b792259818c90a7f478d16be31d306346e0b28b7383 not found: ID does not exist"
Oct 30 14:34:44.718922 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:34:44.718894    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9600-f7a344\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="c3bd77d9-6264-4625-be40-5815c7907002" pod="opendatahub/test-curl-9600-f7a344"
Oct 30 14:34:45.116587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:45.116531    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3bd77d9-6264-4625-be40-5815c7907002" path="/var/lib/kubelet/pods/c3bd77d9-6264-4625-be40-5815c7907002/volumes"
Oct 30 14:34:45.127005 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:34:45.126974    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9600-f7a344\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="c3bd77d9-6264-4625-be40-5815c7907002" pod="opendatahub/test-curl-9600-f7a344"
Oct 30 14:34:45.703271 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:45.703240    4530 generic.go:356] "Generic (PLEG): container finished" podID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerID="16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516" exitCode=0
Oct 30 14:34:45.703453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:45.703318    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" event={"ID":"6bc59ec5-9f0f-481b-8d34-f5e22619f51e","Type":"ContainerDied","Data":"16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516"}
Oct 30 14:34:45.703747 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:45.703725    4530 scope.go:122] "RemoveContainer" containerID="16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516"
Oct 30 14:34:45.705334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:45.705305    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:34:45.705395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:45.705375    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:34:45.705433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:45.705424    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:34:46.717389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:46.717349    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" event={"ID":"6bc59ec5-9f0f-481b-8d34-f5e22619f51e","Type":"ContainerStarted","Data":"036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea"}
Oct 30 14:34:47.343927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:47.343883    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:34:47.344950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:47.344917    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 14:34:47.345125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:47.344980    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 14:34:48.343656 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:48.343617    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 14:34:48.344064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:48.343680    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 14:34:49.341682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:49.341632    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:49.343400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:49.343367    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 14:34:49.343564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:49.343426    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 14:34:49.355350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:49.355328    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:49.753612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:49.753520    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:34:49.983665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:49.983629    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:34:50.344033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:50.343996    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 14:34:50.344198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:50.344054    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 14:34:51.343257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:51.343221    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 14:34:51.343733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:51.343287    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 14:34:52.349013 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:52.348982    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:34:52.350257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:52.350234    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:34:53.498804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.498765    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"]
Oct 30 14:34:53.499381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.499366    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c3bd77d9-6264-4625-be40-5815c7907002" containerName="test-curl-9600-f7a344"
Oct 30 14:34:53.499381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.499380    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3bd77d9-6264-4625-be40-5815c7907002" containerName="test-curl-9600-f7a344"
Oct 30 14:34:53.499533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.499502    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3bd77d9-6264-4625-be40-5815c7907002" containerName="test-curl-9600-f7a344"
Oct 30 14:34:53.504239 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.504215    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.520886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.520852    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"]
Oct 30 14:34:53.530871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.530850    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-228454\"/\"openshift-service-ca.crt\""
Oct 30 14:34:53.531047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.530866    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-228454\"/\"kube-root-ca.crt\""
Oct 30 14:34:53.531047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.530931    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-228454\"/\"default-dockercfg-7hwlw\""
Oct 30 14:34:53.535954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.535937    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-228454\"/\"auth-test-model-228454-kserve-self-signed-certs\""
Oct 30 14:34:53.569319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.569291    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-home\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.569433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.569330    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779351d5-97eb-4384-af2a-293f1f3cbdca-tls-certs\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.569433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.569408    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-dshm\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.569535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.569520    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmvs7\" (UniqueName: \"kubernetes.io/projected/779351d5-97eb-4384-af2a-293f1f3cbdca-kube-api-access-tmvs7\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.569631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.569607    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-model-cache\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.569770 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.569748    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-tmp-dir\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.670607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.670576    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-tmp-dir\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.670769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.670635    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-home\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.670769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.670665    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779351d5-97eb-4384-af2a-293f1f3cbdca-tls-certs\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.670769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.670696    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-dshm\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.670931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.670790    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmvs7\" (UniqueName: \"kubernetes.io/projected/779351d5-97eb-4384-af2a-293f1f3cbdca-kube-api-access-tmvs7\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.670931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.670814    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-model-cache\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.671073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.671039    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-tmp-dir\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.671142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.671064    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-home\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.671301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.671276    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-model-cache\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.673027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.673005    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-dshm\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.673520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.673503    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779351d5-97eb-4384-af2a-293f1f3cbdca-tls-certs\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.697087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.697060    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmvs7\" (UniqueName: \"kubernetes.io/projected/779351d5-97eb-4384-af2a-293f1f3cbdca-kube-api-access-tmvs7\") pod \"auth-test-model-228454-kserve-5c4989b757-dfjlv\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") " pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:53.818555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:53.818514    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:54.127327 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:34:54.127296    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod779351d5_97eb_4384_af2a_293f1f3cbdca.slice/crio-8e84bcc8f226dd70ad5a788825710c698c619ef2ffda0da673bcf0cd3a1547bc WatchSource:0}: Error finding container 8e84bcc8f226dd70ad5a788825710c698c619ef2ffda0da673bcf0cd3a1547bc: Status 404 returned error can't find the container with id 8e84bcc8f226dd70ad5a788825710c698c619ef2ffda0da673bcf0cd3a1547bc
Oct 30 14:34:54.129321 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:54.129294    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"]
Oct 30 14:34:54.783045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:54.783005    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv" event={"ID":"779351d5-97eb-4384-af2a-293f1f3cbdca","Type":"ContainerStarted","Data":"c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b"}
Oct 30 14:34:54.783574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:54.783061    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv" event={"ID":"779351d5-97eb-4384-af2a-293f1f3cbdca","Type":"ContainerStarted","Data":"8e84bcc8f226dd70ad5a788825710c698c619ef2ffda0da673bcf0cd3a1547bc"}
Oct 30 14:34:54.819578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:54.819536    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:34:54.832361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:34:54.832235    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv" podStartSLOduration=1.8322207449999999 podStartE2EDuration="1.832220745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:54.829473127 +0000 UTC m=+2759.861372069" watchObservedRunningTime="2026-10-30 14:34:54.832220745 +0000 UTC m=+2759.864119700"
Oct 30 14:35:03.819186 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:03.819148    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:35:03.831564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:03.831522    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:35:03.863294 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:03.863269    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:35:09.080803 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.080768    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"]
Oct 30 14:35:09.087073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.087051    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.120299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.120271    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-f1f206-openshift-default-dockercfg-kf8p5\""
Oct 30 14:35:09.126644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.126619    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"]
Oct 30 14:35:09.185903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.185429    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-data\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.186450 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.186420    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-credential-socket\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.216616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.197112    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-envoy\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.216616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.197238    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-podinfo\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.216616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.197298    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istiod-ca-cert\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.216616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.197376    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-token\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.216616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.197526    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-socket\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.216616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.197602    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-992r4\" (UniqueName: \"kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-kube-api-access-992r4\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.216616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.197642    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-certs\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.311618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.311570    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-data\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.312413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.312383    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-credential-socket\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.312946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.312926    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-envoy\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.313160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.313144    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-podinfo\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.313300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.313286    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istiod-ca-cert\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.313436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.313422    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-token\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.314436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.312301    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-data\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.314436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.313667    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-socket\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.314436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.313698    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-992r4\" (UniqueName: \"kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-kube-api-access-992r4\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.314436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.313737    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-certs\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.314436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.314129    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-certs\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.314436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.312869    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-credential-socket\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.314900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.314591    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-socket\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.315345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.315324    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istiod-ca-cert\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.319821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.319774    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-envoy\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.322588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.322531    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-podinfo\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.341475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.341241    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-token\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.341475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.341410    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-992r4\" (UniqueName: \"kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-kube-api-access-992r4\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.411955 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.411927    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:09.734463 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:35:09.734431    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc1ef01_f2c5_42ae_9e3d_690cbed016f7.slice/crio-be8bb9d06b029bb43d9ba1b6afe816f2a8c63a4a9a3a65f65af3cb0e16e52fc2 WatchSource:0}: Error finding container be8bb9d06b029bb43d9ba1b6afe816f2a8c63a4a9a3a65f65af3cb0e16e52fc2: Status 404 returned error can't find the container with id be8bb9d06b029bb43d9ba1b6afe816f2a8c63a4a9a3a65f65af3cb0e16e52fc2
Oct 30 14:35:09.736592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.736567    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"]
Oct 30 14:35:09.737988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.737947    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:35:09.738068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.738036    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:35:09.738173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.738087    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:35:09.902610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.902557    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" event={"ID":"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7","Type":"ContainerStarted","Data":"13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d"}
Oct 30 14:35:09.902787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.902673    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" event={"ID":"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7","Type":"ContainerStarted","Data":"be8bb9d06b029bb43d9ba1b6afe816f2a8c63a4a9a3a65f65af3cb0e16e52fc2"}
Oct 30 14:35:09.954512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:09.954459    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" podStartSLOduration=0.954444718 podStartE2EDuration="954.444718ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:35:09.952413971 +0000 UTC m=+2774.984312891" watchObservedRunningTime="2026-10-30 14:35:09.954444718 +0000 UTC m=+2774.986343656"
Oct 30 14:35:10.413002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:10.412963    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:10.420635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:10.420609    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:10.909273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:10.909246    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:10.910025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:10.910008    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:35:11.367338 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:35:11.367302    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:35:11.852325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:11.852264    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"]
Oct 30 14:35:11.856658 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:11.856630    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:11.876139 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:11.876110    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"]
Oct 30 14:35:11.886389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:11.886365    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6e91df\"/\"openshift-service-ca.crt\""
Oct 30 14:35:11.886513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:11.886394    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6e91df\"/\"kube-root-ca.crt\""
Oct 30 14:35:11.888432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:11.888410    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6e91df\"/\"default-dockercfg-xqcdt\""
Oct 30 14:35:11.888685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:11.888655    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6e91df\"/\"sub-test-model-6e91df-kserve-self-signed-certs\""
Oct 30 14:35:12.050615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.050572    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6d75\" (UniqueName: \"kubernetes.io/projected/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-kube-api-access-p6d75\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.050767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.050734    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tls-certs\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.050903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.050882    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tmp-dir\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.050967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.050924    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-home\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.051027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.050972    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-model-cache\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.051192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.051167    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-dshm\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.152718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.152628    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-dshm\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.152961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.152931    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6d75\" (UniqueName: \"kubernetes.io/projected/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-kube-api-access-p6d75\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.153106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.153044    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tls-certs\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.153182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.153134    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tmp-dir\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.153182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.153173    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-home\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.153302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.153205    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-model-cache\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.153717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.153686    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-model-cache\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.154926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.154653    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tmp-dir\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.154926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.154882    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-home\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.157472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.157403    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-dshm\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.157661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.157638    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tls-certs\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.180465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.180440    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6d75\" (UniqueName: \"kubernetes.io/projected/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-kube-api-access-p6d75\") pod \"sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") " pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.472901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.472806    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:12.782707 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:35:12.782678    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3bbdd89_e66c_4a3f_baa5_d1eace65f47c.slice/crio-aeb1191c606d0e1e8edef20bfdddb1490060315ec9ee10e93d292c8d0b04cc5c WatchSource:0}: Error finding container aeb1191c606d0e1e8edef20bfdddb1490060315ec9ee10e93d292c8d0b04cc5c: Status 404 returned error can't find the container with id aeb1191c606d0e1e8edef20bfdddb1490060315ec9ee10e93d292c8d0b04cc5c
Oct 30 14:35:12.785188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.785163    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"]
Oct 30 14:35:12.925393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.925352    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z" event={"ID":"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c","Type":"ContainerStarted","Data":"1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e"}
Oct 30 14:35:12.925798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.925404    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z" event={"ID":"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c","Type":"ContainerStarted","Data":"aeb1191c606d0e1e8edef20bfdddb1490060315ec9ee10e93d292c8d0b04cc5c"}
Oct 30 14:35:12.982021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:12.981973    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z" podStartSLOduration=1.98196024 podStartE2EDuration="1.98196024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:35:12.978338896 +0000 UTC m=+2778.010237852" watchObservedRunningTime="2026-10-30 14:35:12.98196024 +0000 UTC m=+2778.013859178"
Oct 30 14:35:13.473633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:13.473597    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:14.337961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.337928    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"]
Oct 30 14:35:14.342405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.342376    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"
Oct 30 14:35:14.362453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.362398    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"]
Oct 30 14:35:14.370501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.370481    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-f1f206-dockercfg-hpdpd\""
Oct 30 14:35:14.382047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.382023    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk\" (UID: \"61e81b9f-09ff-4a4f-bcef-75c443a1af7f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"
Oct 30 14:35:14.382167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.382151    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znnzj\" (UniqueName: \"kubernetes.io/projected/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-kube-api-access-znnzj\") pod \"payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk\" (UID: \"61e81b9f-09ff-4a4f-bcef-75c443a1af7f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"
Oct 30 14:35:14.395291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.395270    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-f1f206\""
Oct 30 14:35:14.533475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.523684    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk\" (UID: \"61e81b9f-09ff-4a4f-bcef-75c443a1af7f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"
Oct 30 14:35:14.533475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.525010    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk\" (UID: \"61e81b9f-09ff-4a4f-bcef-75c443a1af7f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"
Oct 30 14:35:14.533475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.525317    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znnzj\" (UniqueName: \"kubernetes.io/projected/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-kube-api-access-znnzj\") pod \"payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk\" (UID: \"61e81b9f-09ff-4a4f-bcef-75c443a1af7f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"
Oct 30 14:35:14.558039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.556485    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znnzj\" (UniqueName: \"kubernetes.io/projected/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-kube-api-access-znnzj\") pod \"payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk\" (UID: \"61e81b9f-09ff-4a4f-bcef-75c443a1af7f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"
Oct 30 14:35:14.658449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.658373    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"
Oct 30 14:35:14.962883 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:35:14.962856    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e81b9f_09ff_4a4f_bcef_75c443a1af7f.slice/crio-4c3844e73cb766efc1e76944a57e9509c67c0c832a8f681e48cdd25c05e6de2b WatchSource:0}: Error finding container 4c3844e73cb766efc1e76944a57e9509c67c0c832a8f681e48cdd25c05e6de2b: Status 404 returned error can't find the container with id 4c3844e73cb766efc1e76944a57e9509c67c0c832a8f681e48cdd25c05e6de2b
Oct 30 14:35:14.963810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:14.963787    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"]
Oct 30 14:35:15.950049 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:15.950018    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk" event={"ID":"61e81b9f-09ff-4a4f-bcef-75c443a1af7f","Type":"ContainerStarted","Data":"c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c"}
Oct 30 14:35:15.950490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:15.950067    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk" event={"ID":"61e81b9f-09ff-4a4f-bcef-75c443a1af7f","Type":"ContainerStarted","Data":"4c3844e73cb766efc1e76944a57e9509c67c0c832a8f681e48cdd25c05e6de2b"}
Oct 30 14:35:15.950490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:15.950159    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"
Oct 30 14:35:16.000484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:16.000433    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk" podStartSLOduration=2.000421385 podStartE2EDuration="2.000421385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:35:15.995953575 +0000 UTC m=+2781.027852514" watchObservedRunningTime="2026-10-30 14:35:16.000421385 +0000 UTC m=+2781.032320325"
Oct 30 14:35:18.273786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:18.273747    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:35:22.346527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:22.346484    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"]
Oct 30 14:35:22.347252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:22.347215    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy" containerID="cri-o://52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e" gracePeriod=30
Oct 30 14:35:22.357129 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:22.357104    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"]
Oct 30 14:35:22.485567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:22.485220    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:22.514980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:22.514947    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:23.025104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:23.025076    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:35:26.065821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:26.065786    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"]
Oct 30 14:35:26.961747 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:26.961719    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"
Oct 30 14:35:27.641528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.641502    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:35:27.740336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.740296    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-kube-api-access-7v7jx\" (UniqueName: \"kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-kube-api-access-7v7jx\") pod \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") "
Oct 30 14:35:27.740601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.740384    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-data\") pod \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") "
Oct 30 14:35:27.740601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.740454    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-podinfo\") pod \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") "
Oct 30 14:35:27.740601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.740571    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-token\" (UniqueName: \"kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-token\") pod \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") "
Oct 30 14:35:27.740800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.740611    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-socket\") pod \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") "
Oct 30 14:35:27.740800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.740683    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-certs\") pod \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") "
Oct 30 14:35:27.740800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.740756    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-credential-socket\") pod \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") "
Oct 30 14:35:27.741034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.740823    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-envoy\") pod \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") "
Oct 30 14:35:27.741034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.740907    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istiod-ca-cert\") pod \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\" (UID: \"7a91e56c-d22c-40a4-a8f1-e144eece82d4\") "
Oct 30 14:35:27.743407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.741967    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-credential-socket" pod "7a91e56c-d22c-40a4-a8f1-e144eece82d4" (UID: "7a91e56c-d22c-40a4-a8f1-e144eece82d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:27.743407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.743174    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istiod-ca-cert" pod "7a91e56c-d22c-40a4-a8f1-e144eece82d4" (UID: "7a91e56c-d22c-40a4-a8f1-e144eece82d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:35:27.743407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.743270    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-socket" pod "7a91e56c-d22c-40a4-a8f1-e144eece82d4" (UID: "7a91e56c-d22c-40a4-a8f1-e144eece82d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:27.743407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.743328    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-data" pod "7a91e56c-d22c-40a4-a8f1-e144eece82d4" (UID: "7a91e56c-d22c-40a4-a8f1-e144eece82d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:27.744370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.744346    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-certs" pod "7a91e56c-d22c-40a4-a8f1-e144eece82d4" (UID: "7a91e56c-d22c-40a4-a8f1-e144eece82d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:27.744607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.744383    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-podinfo" pod "7a91e56c-d22c-40a4-a8f1-e144eece82d4" (UID: "7a91e56c-d22c-40a4-a8f1-e144eece82d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:35:27.744607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.744560    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-envoy" pod "7a91e56c-d22c-40a4-a8f1-e144eece82d4" (UID: "7a91e56c-d22c-40a4-a8f1-e144eece82d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:27.744930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.744904    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-token" pod "7a91e56c-d22c-40a4-a8f1-e144eece82d4" (UID: "7a91e56c-d22c-40a4-a8f1-e144eece82d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:35:27.746302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.746281    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-kube-api-access-7v7jx" pod "7a91e56c-d22c-40a4-a8f1-e144eece82d4" (UID: "7a91e56c-d22c-40a4-a8f1-e144eece82d4"). InnerVolumeSpecName "kube-api-access-7v7jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:35:27.842643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.842613    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:27.842643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.842640    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v7jx\" (UniqueName: \"kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-kube-api-access-7v7jx\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:27.842810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.842649    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:27.842810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.842658    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:27.842810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.842666    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:27.842810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.842675    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:27.842810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.842683    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:27.842810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.842693    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:27.842810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:27.842702    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a91e56c-d22c-40a4-a8f1-e144eece82d4-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:28.060154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:28.060123    4530 generic.go:356] "Generic (PLEG): container finished" podID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerID="52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e" exitCode=0
Oct 30 14:35:28.060325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:28.060178    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"
Oct 30 14:35:28.060325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:28.060177    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" event={"ID":"7a91e56c-d22c-40a4-a8f1-e144eece82d4","Type":"ContainerDied","Data":"52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e"}
Oct 30 14:35:28.060325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:28.060246    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w" event={"ID":"7a91e56c-d22c-40a4-a8f1-e144eece82d4","Type":"ContainerDied","Data":"3a1efd0b76abc881886e99cbe25d5e4136e9b88612b1350078ef674532dcf9af"}
Oct 30 14:35:28.060325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:28.060275    4530 scope.go:122] "RemoveContainer" containerID="52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e"
Oct 30 14:35:28.073914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:28.073889    4530 scope.go:122] "RemoveContainer" containerID="6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607"
Oct 30 14:35:28.084747 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:28.084730    4530 scope.go:122] "RemoveContainer" containerID="52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e"
Oct 30 14:35:28.085026 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:35:28.085006    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e\": container with ID starting with 52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e not found: ID does not exist" containerID="52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e"
Oct 30 14:35:28.085078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:28.085037    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e"} err="failed to get container status \"52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e\": rpc error: code = NotFound desc = could not find container \"52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e\": container with ID starting with 52908592f3d09ff889ba30443276cbdeb68f5c3fb4fa642d0c6511f4f10d8a4e not found: ID does not exist"
Oct 30 14:35:28.085078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:28.085053    4530 scope.go:122] "RemoveContainer" containerID="6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607"
Oct 30 14:35:28.085286 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:35:28.085268    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607\": container with ID starting with 6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607 not found: ID does not exist" containerID="6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607"
Oct 30 14:35:28.085336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:28.085292    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607"} err="failed to get container status \"6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607\": rpc error: code = NotFound desc = could not find container \"6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607\": container with ID starting with 6ab0b8d491346bd1d2af2f552ee968620e3719d090f3c7225e5daba2551a7607 not found: ID does not exist"
Oct 30 14:35:28.124939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:28.124917    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"]
Oct 30 14:35:28.130066 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:28.130046    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-ljm4w"]
Oct 30 14:35:29.115232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:29.115196    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" path="/var/lib/kubelet/pods/7a91e56c-d22c-40a4-a8f1-e144eece82d4/volumes"
Oct 30 14:35:33.653520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.653481    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"]
Oct 30 14:35:33.654562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.654524    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy"
Oct 30 14:35:33.654650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.654567    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy"
Oct 30 14:35:33.654650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.654633    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy"
Oct 30 14:35:33.654650 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.654642    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy"
Oct 30 14:35:33.654825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.654810    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy"
Oct 30 14:35:33.660778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.660756    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.678673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.678644    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"]
Oct 30 14:35:33.681942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.681907    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-c7c37b-openshift-default-dockercfg-xbblf\""
Oct 30 14:35:33.716633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.716597    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-data\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.716779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.716649    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-token\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.716779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.716669    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2abb44c-4a6d-44b2-8081-d24b067dda00-istiod-ca-cert\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.716931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.716796    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-envoy\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.716931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.716910    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-certs\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.717055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.716954    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l2pq\" (UniqueName: \"kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-kube-api-access-5l2pq\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.717112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.717053    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-socket\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.717112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.717085    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-podinfo\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.717295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.717277    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-credential-socket\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.823560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.822608    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-certs\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.823560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.822669    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l2pq\" (UniqueName: \"kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-kube-api-access-5l2pq\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.823560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.822717    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-socket\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.823560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.822744    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-podinfo\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.823560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.822880    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-credential-socket\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.823560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.822984    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-data\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.823560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.823030    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-token\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.823560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.823055    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2abb44c-4a6d-44b2-8081-d24b067dda00-istiod-ca-cert\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.823560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.823098    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-envoy\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.841220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.841182    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-podinfo\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.841495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.841473    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-certs\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.842045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.842020    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-socket\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.843107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.842442    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-data\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.843107 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.842700    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-credential-socket\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.843662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.843569    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2abb44c-4a6d-44b2-8081-d24b067dda00-istiod-ca-cert\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.853136 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.853110    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-envoy\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.889427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.888858    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l2pq\" (UniqueName: \"kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-kube-api-access-5l2pq\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.889952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.889835    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-token\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:33.978197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:33.978095    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:34.318787 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:35:34.318757    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2abb44c_4a6d_44b2_8081_d24b067dda00.slice/crio-c6d113bc7bdbfef329d30d4cd47f782fc3f516f8f2d13d2b34f4f1d43e5ad957 WatchSource:0}: Error finding container c6d113bc7bdbfef329d30d4cd47f782fc3f516f8f2d13d2b34f4f1d43e5ad957: Status 404 returned error can't find the container with id c6d113bc7bdbfef329d30d4cd47f782fc3f516f8f2d13d2b34f4f1d43e5ad957
Oct 30 14:35:34.320745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:34.320718    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"]
Oct 30 14:35:34.321576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:34.321523    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:35:34.321632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:34.321610    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:35:34.321670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:34.321639    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:35:35.125194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:35.125153    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" event={"ID":"b2abb44c-4a6d-44b2-8081-d24b067dda00","Type":"ContainerStarted","Data":"184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472"}
Oct 30 14:35:35.125606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:35.125218    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" event={"ID":"b2abb44c-4a6d-44b2-8081-d24b067dda00","Type":"ContainerStarted","Data":"c6d113bc7bdbfef329d30d4cd47f782fc3f516f8f2d13d2b34f4f1d43e5ad957"}
Oct 30 14:35:35.202862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:35.202813    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" podStartSLOduration=2.202799395 podStartE2EDuration="2.202799395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:35:35.188191749 +0000 UTC m=+2800.220090691" watchObservedRunningTime="2026-10-30 14:35:35.202799395 +0000 UTC m=+2800.234698333"
Oct 30 14:35:35.978925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:35.978892    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:35.984587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:35.984559    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:36.132054 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:36.132022    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:36.133028 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:36.133005    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:35:37.385257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.385218    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx" podUID="6f78d99a-71a0-4ed7-88fb-7fc54fada30d" containerName="main" containerID="cri-o://58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff" gracePeriod=45
Oct 30 14:35:37.560188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.560145    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx" podUID="6f78d99a-71a0-4ed7-88fb-7fc54fada30d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.251:8000/ready\": dial tcp 10.133.0.251:8000: connect: connection refused"
Oct 30 14:35:37.693520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.693490    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:35:37.774787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.774751    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-model-cache\") pod \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") "
Oct 30 14:35:37.774980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.774812    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-kube-api-access-6b2cr\" (UniqueName: \"kubernetes.io/projected/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-kube-api-access-6b2cr\") pod \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") "
Oct 30 14:35:37.774980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.774926    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tmp-dir\") pod \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") "
Oct 30 14:35:37.775110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.775035    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-dshm\") pod \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") "
Oct 30 14:35:37.775110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.775052    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-model-cache" pod "6f78d99a-71a0-4ed7-88fb-7fc54fada30d" (UID: "6f78d99a-71a0-4ed7-88fb-7fc54fada30d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:37.775232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.775152    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-home\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-home\") pod \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") "
Oct 30 14:35:37.775232 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.775185    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tls-certs\") pod \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\" (UID: \"6f78d99a-71a0-4ed7-88fb-7fc54fada30d\") "
Oct 30 14:35:37.775654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.775630    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-home" pod "6f78d99a-71a0-4ed7-88fb-7fc54fada30d" (UID: "6f78d99a-71a0-4ed7-88fb-7fc54fada30d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:37.775906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.775887    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:37.776039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.776020    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tmp-dir" pod "6f78d99a-71a0-4ed7-88fb-7fc54fada30d" (UID: "6f78d99a-71a0-4ed7-88fb-7fc54fada30d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:37.776125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.776039    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:37.777716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.777689    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-dshm" pod "6f78d99a-71a0-4ed7-88fb-7fc54fada30d" (UID: "6f78d99a-71a0-4ed7-88fb-7fc54fada30d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:37.777820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.777770    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-kube-api-access-6b2cr" pod "6f78d99a-71a0-4ed7-88fb-7fc54fada30d" (UID: "6f78d99a-71a0-4ed7-88fb-7fc54fada30d"). InnerVolumeSpecName "kube-api-access-6b2cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:35:37.778116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.778090    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tls-certs" pod "6f78d99a-71a0-4ed7-88fb-7fc54fada30d" (UID: "6f78d99a-71a0-4ed7-88fb-7fc54fada30d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:35:37.877805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.877782    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:37.877805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.877803    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:37.877960 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.877814    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b2cr\" (UniqueName: \"kubernetes.io/projected/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-kube-api-access-6b2cr\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:37.877960 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:37.877840    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f78d99a-71a0-4ed7-88fb-7fc54fada30d-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:38.076526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:38.076494    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"]
Oct 30 14:35:38.184384 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:38.184347    4530 generic.go:356] "Generic (PLEG): container finished" podID="6f78d99a-71a0-4ed7-88fb-7fc54fada30d" containerID="58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff" exitCode=0
Oct 30 14:35:38.184624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:38.184396    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"
Oct 30 14:35:38.184624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:38.184430    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx" event={"ID":"6f78d99a-71a0-4ed7-88fb-7fc54fada30d","Type":"ContainerDied","Data":"58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff"}
Oct 30 14:35:38.184624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:38.184477    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx" event={"ID":"6f78d99a-71a0-4ed7-88fb-7fc54fada30d","Type":"ContainerDied","Data":"74ec0187af3f5f3f230bb6ed1308668209f474a77bfe8a3e9e926717eb7b3a5d"}
Oct 30 14:35:38.184624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:38.184494    4530 scope.go:122] "RemoveContainer" containerID="58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff"
Oct 30 14:35:38.204103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:38.204080    4530 scope.go:122] "RemoveContainer" containerID="58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff"
Oct 30 14:35:38.204668 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:35:38.204601    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff\": container with ID starting with 58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff not found: ID does not exist" containerID="58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff"
Oct 30 14:35:38.204668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:38.204639    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff"} err="failed to get container status \"58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff\": rpc error: code = NotFound desc = could not find container \"58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff\": container with ID starting with 58ef7f2a90f0a5c99a046c76c1a914a6c97974470401af3a75310a53c22492ff not found: ID does not exist"
Oct 30 14:35:38.249473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:38.249447    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"]
Oct 30 14:35:38.257477 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:38.257445    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a59ecb/auth-test-model-a59ecb-kserve-85469cf445-rvqpx"]
Oct 30 14:35:39.115360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:39.115325    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f78d99a-71a0-4ed7-88fb-7fc54fada30d" path="/var/lib/kubelet/pods/6f78d99a-71a0-4ed7-88fb-7fc54fada30d/volumes"
Oct 30 14:35:41.087178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.087135    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv" podUID="e0c01eea-69d5-4c86-bc41-ccd1143c36e7" containerName="main" containerID="cri-o://677633d62f6d0dee313a5b7ff71f2cc8956cdc2839d6d5e5ad0dc108b8063e75" gracePeriod=45
Oct 30 14:35:41.208275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.208250    4530 generic.go:356] "Generic (PLEG): container finished" podID="e0c01eea-69d5-4c86-bc41-ccd1143c36e7" containerID="677633d62f6d0dee313a5b7ff71f2cc8956cdc2839d6d5e5ad0dc108b8063e75" exitCode=0
Oct 30 14:35:41.208378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.208319    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv" event={"ID":"e0c01eea-69d5-4c86-bc41-ccd1143c36e7","Type":"ContainerDied","Data":"677633d62f6d0dee313a5b7ff71f2cc8956cdc2839d6d5e5ad0dc108b8063e75"}
Oct 30 14:35:41.359072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.359047    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:35:41.367824 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:35:41.367789    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:35:41.429705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.429670    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tls-certs\") pod \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") "
Oct 30 14:35:41.429892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.429873    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tmp-dir\") pod \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") "
Oct 30 14:35:41.429992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.429911    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-kube-api-access-klhnm\" (UniqueName: \"kubernetes.io/projected/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-kube-api-access-klhnm\") pod \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") "
Oct 30 14:35:41.429992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.429944    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-dshm\") pod \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") "
Oct 30 14:35:41.430124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.430008    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-home\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-home\") pod \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") "
Oct 30 14:35:41.430124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.430069    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-model-cache\") pod \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\" (UID: \"e0c01eea-69d5-4c86-bc41-ccd1143c36e7\") "
Oct 30 14:35:41.430244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.430177    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tmp-dir" pod "e0c01eea-69d5-4c86-bc41-ccd1143c36e7" (UID: "e0c01eea-69d5-4c86-bc41-ccd1143c36e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:41.430663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.430627    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-model-cache" pod "e0c01eea-69d5-4c86-bc41-ccd1143c36e7" (UID: "e0c01eea-69d5-4c86-bc41-ccd1143c36e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:41.430797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.430709    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:41.431171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.431130    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-home" pod "e0c01eea-69d5-4c86-bc41-ccd1143c36e7" (UID: "e0c01eea-69d5-4c86-bc41-ccd1143c36e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:41.432687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.432653    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-dshm" pod "e0c01eea-69d5-4c86-bc41-ccd1143c36e7" (UID: "e0c01eea-69d5-4c86-bc41-ccd1143c36e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:41.434581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.434359    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-kube-api-access-klhnm" pod "e0c01eea-69d5-4c86-bc41-ccd1143c36e7" (UID: "e0c01eea-69d5-4c86-bc41-ccd1143c36e7"). InnerVolumeSpecName "kube-api-access-klhnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:35:41.434581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.434414    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tls-certs" pod "e0c01eea-69d5-4c86-bc41-ccd1143c36e7" (UID: "e0c01eea-69d5-4c86-bc41-ccd1143c36e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:35:41.532036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.532010    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klhnm\" (UniqueName: \"kubernetes.io/projected/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-kube-api-access-klhnm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:41.532036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.532035    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:41.532189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.532044    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:41.532189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.532052    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:41.532189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:41.532062    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c01eea-69d5-4c86-bc41-ccd1143c36e7-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:42.216298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:42.216267    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"
Oct 30 14:35:42.216808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:42.216297    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv" event={"ID":"e0c01eea-69d5-4c86-bc41-ccd1143c36e7","Type":"ContainerDied","Data":"9382d9708e6005e8dd247e06bea8c6daf48c237eb66defe9ac2fffb2b3d2bc48"}
Oct 30 14:35:42.216808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:42.216348    4530 scope.go:122] "RemoveContainer" containerID="677633d62f6d0dee313a5b7ff71f2cc8956cdc2839d6d5e5ad0dc108b8063e75"
Oct 30 14:35:42.277341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:42.277317    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"]
Oct 30 14:35:42.291555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:42.291516    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-acefbb/auth-test-model-acefbb-kserve-84b9c64f5c-d4xrv"]
Oct 30 14:35:43.115703 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:43.115669    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c01eea-69d5-4c86-bc41-ccd1143c36e7" path="/var/lib/kubelet/pods/e0c01eea-69d5-4c86-bc41-ccd1143c36e7/volumes"
Oct 30 14:35:47.901250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:47.901215    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"]
Oct 30 14:35:47.902024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:47.902003    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e0c01eea-69d5-4c86-bc41-ccd1143c36e7" containerName="main"
Oct 30 14:35:47.902024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:47.902025    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c01eea-69d5-4c86-bc41-ccd1143c36e7" containerName="main"
Oct 30 14:35:47.902126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:47.902061    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6f78d99a-71a0-4ed7-88fb-7fc54fada30d" containerName="main"
Oct 30 14:35:47.902126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:47.902067    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f78d99a-71a0-4ed7-88fb-7fc54fada30d" containerName="main"
Oct 30 14:35:47.902197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:47.902157    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f78d99a-71a0-4ed7-88fb-7fc54fada30d" containerName="main"
Oct 30 14:35:47.902197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:47.902169    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a91e56c-d22c-40a4-a8f1-e144eece82d4" containerName="istio-proxy"
Oct 30 14:35:47.902197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:47.902176    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c01eea-69d5-4c86-bc41-ccd1143c36e7" containerName="main"
Oct 30 14:35:47.905706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:47.905688    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:47.937745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:47.937718    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-228454-openshift-default-dockercfg-nlnt2\""
Oct 30 14:35:47.942977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:47.942950    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"]
Oct 30 14:35:48.014486 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.014453    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-token\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.014640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.014489    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmm25\" (UniqueName: \"kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-kube-api-access-lmm25\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.014733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.014628    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istiod-ca-cert\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.014811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.014763    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-podinfo\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.014861 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.014841    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-credential-socket\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.015010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.014987    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-certs\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.015081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.015030    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-socket\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.015145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.015114    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-data\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.015202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.015149    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-envoy\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.116512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.116481    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-podinfo\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.116512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.116517    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-credential-socket\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.116746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.116612    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-certs\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.116746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.116643    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-socket\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.116746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.116686    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-data\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.116746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.116716    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-envoy\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.116959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.116786    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-token\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.116959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.116809    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmm25\" (UniqueName: \"kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-kube-api-access-lmm25\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.116959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.116863    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istiod-ca-cert\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.117152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.117056    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-certs\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.117152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.117105    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-credential-socket\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.117301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.117167    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-socket\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.117533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.117505    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-data\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.117845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.117822    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istiod-ca-cert\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.119140 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.119118    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-envoy\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.119241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.119226    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-podinfo\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.143309 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.143285    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-token\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.143535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.143516    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmm25\" (UniqueName: \"kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-kube-api-access-lmm25\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.220698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.220639    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:48.541166 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:35:48.541138    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc263df1_ccb8_44d7_9f21_a065c9aa64c0.slice/crio-a9820ecc6c2ed39f22341f9b6351d297e3e748adf91d64b4fed317d2ed552c99 WatchSource:0}: Error finding container a9820ecc6c2ed39f22341f9b6351d297e3e748adf91d64b4fed317d2ed552c99: Status 404 returned error can't find the container with id a9820ecc6c2ed39f22341f9b6351d297e3e748adf91d64b4fed317d2ed552c99
Oct 30 14:35:48.542264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.542242    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"]
Oct 30 14:35:48.543965 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.543930    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:35:48.544048 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.543994    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:35:48.544048 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.544026    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:35:48.762181 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.762148    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"]
Oct 30 14:35:48.766400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.766382    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:48.795597 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.792155    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-f1f206-serving-cert\""
Oct 30 14:35:48.814669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.814631    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"]
Oct 30 14:35:48.829728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.829608    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:48.830189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.830100    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:48.830189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.830162    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb924\" (UniqueName: \"kubernetes.io/projected/ad23228d-09a6-4eb9-a818-746f037fa303-kube-api-access-lb924\") pod \"maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:48.830359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.830329    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad23228d-09a6-4eb9-a818-746f037fa303-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:48.931761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.931728    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad23228d-09a6-4eb9-a818-746f037fa303-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:48.932287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.931833    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:48.932287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.931977    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:48.932287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.932024    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb924\" (UniqueName: \"kubernetes.io/projected/ad23228d-09a6-4eb9-a818-746f037fa303-kube-api-access-lb924\") pod \"maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:48.932500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.932474    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad23228d-09a6-4eb9-a818-746f037fa303-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:48.934440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.934418    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:48.934440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.934434    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:48.988068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:48.988029    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb924\" (UniqueName: \"kubernetes.io/projected/ad23228d-09a6-4eb9-a818-746f037fa303-kube-api-access-lb924\") pod \"maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:49.092987 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.092953    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:49.124570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.123871    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"]
Oct 30 14:35:49.129584 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.129524    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.138577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.138521    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"]
Oct 30 14:35:49.149586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.149045    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d3a829-openshift-default-dockercfg-94hwc\""
Oct 30 14:35:49.244576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.242534    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-data\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.244576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.242672    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-podinfo\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.244576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.242702    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istiod-ca-cert\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.244576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.242762    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxlf8\" (UniqueName: \"kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-kube-api-access-dxlf8\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.244576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.242802    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-envoy\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.244576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.242864    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-socket\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.244576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.242991    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-token\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.244576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.243048    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-certs\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.244576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.243134    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-credential-socket\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.362652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.356832    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-data\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.362652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.356960    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-podinfo\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.362652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.356986    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istiod-ca-cert\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.362652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.357029    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxlf8\" (UniqueName: \"kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-kube-api-access-dxlf8\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.362652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.357066    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-envoy\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.362652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.357319    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-socket\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.362652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.358005    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-token\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.362652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.358062    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-certs\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.362652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.358144    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-credential-socket\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.362652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.360754    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-credential-socket\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.362652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.361027    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-data\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.364200 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.363793    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-socket\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.364735 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.364699    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-certs\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.365746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.365718    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istiod-ca-cert\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.367087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.367062    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s" event={"ID":"dc263df1-ccb8-44d7-9f21-a065c9aa64c0","Type":"ContainerStarted","Data":"df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423"}
Oct 30 14:35:49.367214 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.367119    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s" event={"ID":"dc263df1-ccb8-44d7-9f21-a065c9aa64c0","Type":"ContainerStarted","Data":"a9820ecc6c2ed39f22341f9b6351d297e3e748adf91d64b4fed317d2ed552c99"}
Oct 30 14:35:49.371609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.371531    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-podinfo\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.375381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.375358    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-envoy\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.391366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.391297    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-token\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.392244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.392213    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxlf8\" (UniqueName: \"kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-kube-api-access-dxlf8\") pod \"e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") " pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.425037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.424976    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s" podStartSLOduration=2.424959228 podStartE2EDuration="2.424959228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:35:49.422110048 +0000 UTC m=+2814.454008998" watchObservedRunningTime="2026-10-30 14:35:49.424959228 +0000 UTC m=+2814.456858166"
Oct 30 14:35:49.447201 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:35:49.447168    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad23228d_09a6_4eb9_a818_746f037fa303.slice/crio-f640347246156b10a6aad4827516ecf3a22f99ab6b47df9e07f944fd694a1e0c WatchSource:0}: Error finding container f640347246156b10a6aad4827516ecf3a22f99ab6b47df9e07f944fd694a1e0c: Status 404 returned error can't find the container with id f640347246156b10a6aad4827516ecf3a22f99ab6b47df9e07f944fd694a1e0c
Oct 30 14:35:49.447922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.447898    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"]
Oct 30 14:35:49.449143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.449123    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:35:49.463422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.463398    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:49.815336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.811535    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"]
Oct 30 14:35:49.815336 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:35:49.814048    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330c4039_1ce9_4eee_af89_a18cbc37e9a0.slice/crio-da15264a53ca7280d59f8053a64a6c20f3c87bf64a0ad731e915b5a342b98640 WatchSource:0}: Error finding container da15264a53ca7280d59f8053a64a6c20f3c87bf64a0ad731e915b5a342b98640: Status 404 returned error can't find the container with id da15264a53ca7280d59f8053a64a6c20f3c87bf64a0ad731e915b5a342b98640
Oct 30 14:35:49.817522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.817480    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:35:49.817654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.817572    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:35:49.817654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:49.817605    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:35:50.225621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:50.224868    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:50.231561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:50.229040    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 14:35:50.231561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:50.229100    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s" podUID="dc263df1-ccb8-44d7-9f21-a065c9aa64c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 14:35:50.377560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:50.377504    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5" event={"ID":"ad23228d-09a6-4eb9-a818-746f037fa303","Type":"ContainerStarted","Data":"a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e"}
Oct 30 14:35:50.377760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:50.377591    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5" event={"ID":"ad23228d-09a6-4eb9-a818-746f037fa303","Type":"ContainerStarted","Data":"f640347246156b10a6aad4827516ecf3a22f99ab6b47df9e07f944fd694a1e0c"}
Oct 30 14:35:50.377760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:50.377628    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:35:50.385563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:50.380174    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" event={"ID":"330c4039-1ce9-4eee-af89-a18cbc37e9a0","Type":"ContainerStarted","Data":"631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0"}
Oct 30 14:35:50.385563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:50.380225    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" event={"ID":"330c4039-1ce9-4eee-af89-a18cbc37e9a0","Type":"ContainerStarted","Data":"da15264a53ca7280d59f8053a64a6c20f3c87bf64a0ad731e915b5a342b98640"}
Oct 30 14:35:50.434252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:50.434190    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5" podStartSLOduration=2.019993123 podStartE2EDuration="2.434172446s" totalImagesPullingTime="414.179323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:35:50.433732097 +0000 UTC m=+2815.465631037" watchObservedRunningTime="2026-10-30 14:35:50.434172446 +0000 UTC m=+2815.466071386"
Oct 30 14:35:50.463885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:50.463856    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:50.469896 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:50.469872    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:50.504615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:50.503763    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" podStartSLOduration=1.503742822 podStartE2EDuration="1.503742822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:35:50.500920548 +0000 UTC m=+2815.532819491" watchObservedRunningTime="2026-10-30 14:35:50.503742822 +0000 UTC m=+2815.535641763"
Oct 30 14:35:51.221667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:51.221630    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 14:35:51.221852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:51.221685    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s" podUID="dc263df1-ccb8-44d7-9f21-a065c9aa64c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 14:35:51.410365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:51.410317    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:51.411200 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:51.411167    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:35:52.221220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:52.221188    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 14:35:52.221416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:52.221253    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s" podUID="dc263df1-ccb8-44d7-9f21-a065c9aa64c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 14:35:53.102692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.102651    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2" podUID="8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" containerName="main" containerID="cri-o://3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf" gracePeriod=45
Oct 30 14:35:53.221030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.220987    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 14:35:53.221211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.221043    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s" podUID="dc263df1-ccb8-44d7-9f21-a065c9aa64c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 14:35:53.257790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.257751    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2" podUID="8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.250:8000/ready\": dial tcp 10.133.0.250:8000: connect: connection refused"
Oct 30 14:35:53.430576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.430525    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:35:53.435582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.435518    4530 generic.go:356] "Generic (PLEG): container finished" podID="8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" containerID="3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf" exitCode=0
Oct 30 14:35:53.435745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.435665    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2" event={"ID":"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9","Type":"ContainerDied","Data":"3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf"}
Oct 30 14:35:53.435745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.435701    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"
Oct 30 14:35:53.435745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.435711    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2" event={"ID":"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9","Type":"ContainerDied","Data":"ea27ae4f50d224caa6c9f996178c6e8d7738c11855b2f519513a299fa87df79a"}
Oct 30 14:35:53.435872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.435834    4530 scope.go:122] "RemoveContainer" containerID="3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf"
Oct 30 14:35:53.447454 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.447438    4530 scope.go:122] "RemoveContainer" containerID="3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf"
Oct 30 14:35:53.447722 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:35:53.447700    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf\": container with ID starting with 3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf not found: ID does not exist" containerID="3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf"
Oct 30 14:35:53.447822 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.447734    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf"} err="failed to get container status \"3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf\": rpc error: code = NotFound desc = could not find container \"3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf\": container with ID starting with 3c0578323c6c1cb124e6423dd2fdedc55146d17e64ed386ea3a43809a1a78ecf not found: ID does not exist"
Oct 30 14:35:53.536288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.536258    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tmp-dir\") pod \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") "
Oct 30 14:35:53.536514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.536315    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-dshm\") pod \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") "
Oct 30 14:35:53.536514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.536368    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-kube-api-access-cpgqk\" (UniqueName: \"kubernetes.io/projected/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-kube-api-access-cpgqk\") pod \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") "
Oct 30 14:35:53.536514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.536452    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-model-cache\") pod \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") "
Oct 30 14:35:53.536514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.536484    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-home\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-home\") pod \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") "
Oct 30 14:35:53.536820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.536519    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tls-certs\") pod \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\" (UID: \"8979a6ea-2a77-43e3-8c28-157cf5b4f0f9\") "
Oct 30 14:35:53.536820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.536616    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tmp-dir" pod "8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" (UID: "8979a6ea-2a77-43e3-8c28-157cf5b4f0f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:53.536965 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.536806    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-home" pod "8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" (UID: "8979a6ea-2a77-43e3-8c28-157cf5b4f0f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:53.537376 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.537318    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:53.537570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.537531    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:53.537570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.537425    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-model-cache" pod "8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" (UID: "8979a6ea-2a77-43e3-8c28-157cf5b4f0f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:53.539152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.539127    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-kube-api-access-cpgqk" pod "8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" (UID: "8979a6ea-2a77-43e3-8c28-157cf5b4f0f9"). InnerVolumeSpecName "kube-api-access-cpgqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:35:53.539654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.539625    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-dshm" pod "8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" (UID: "8979a6ea-2a77-43e3-8c28-157cf5b4f0f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:35:53.539768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.539648    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tls-certs" pod "8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" (UID: "8979a6ea-2a77-43e3-8c28-157cf5b4f0f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:35:53.639275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.639200    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpgqk\" (UniqueName: \"kubernetes.io/projected/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-kube-api-access-cpgqk\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:53.639275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.639232    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:53.639275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.639242    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:53.639275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.639250    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:35:53.809588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.809374    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"]
Oct 30 14:35:53.814173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:53.814145    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-1d6347/auth-test-model-1d6347-kserve-6fbd7bb6b5-54hr2"]
Oct 30 14:35:54.221224 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:54.221188    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 14:35:54.221652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:54.221247    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s" podUID="dc263df1-ccb8-44d7-9f21-a065c9aa64c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 14:35:55.117188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:55.117153    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" path="/var/lib/kubelet/pods/8979a6ea-2a77-43e3-8c28-157cf5b4f0f9/volumes"
Oct 30 14:35:55.221473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:55.221420    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 14:35:55.222117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:55.221502    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s" podUID="dc263df1-ccb8-44d7-9f21-a065c9aa64c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 14:35:56.226161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:56.226130    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:56.226582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:56.226563    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:56.227135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:56.227118    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:35:56.415375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:35:56.415347    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:36:11.367429 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:36:11.367344    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:36:15.964949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:15.964912    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"]
Oct 30 14:36:15.965604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:15.965569    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" containerName="main"
Oct 30 14:36:15.965604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:15.965586    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" containerName="main"
Oct 30 14:36:15.965712 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:15.965707    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8979a6ea-2a77-43e3-8c28-157cf5b4f0f9" containerName="main"
Oct 30 14:36:15.970728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:15.970696    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:15.995456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:15.994561    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"]
Oct 30 14:36:15.996186 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:15.996126    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-c7c37b-serving-cert\""
Oct 30 14:36:16.095792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.095762    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.095792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.095796    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.096015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.095889    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.096063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.096022    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvm9q\" (UniqueName: \"kubernetes.io/projected/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-kube-api-access-lvm9q\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.205567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.199526    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvm9q\" (UniqueName: \"kubernetes.io/projected/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-kube-api-access-lvm9q\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.205567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.199836    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.205567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.199867    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.205567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.199985    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.205567 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:36:16.200255    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-serving-cert: secret "maas-api-e2e-shared-a-w1-c7c37b-serving-cert" not found
Oct 30 14:36:16.205567 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:36:16.200337    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-maas-api-tls podName:f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d nodeName:}" failed. No retries permitted until 2026-10-30 14:36:16.700315765 +0000 UTC m=+2841.732214706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-maas-api-tls") pod "maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq" (UID: "f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d") : secret "maas-api-e2e-shared-a-w1-c7c37b-serving-cert" not found
Oct 30 14:36:16.206188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.206162    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.217460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.217388    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.233152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.233126    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvm9q\" (UniqueName: \"kubernetes.io/projected/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-kube-api-access-lvm9q\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.706613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.706566    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.708998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.708970    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:16.885950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:16.885917    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:17.223568 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:36:17.222089    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dd8280_f315_4321_a5ab_fd2f1a8a9b6d.slice/crio-fd2b6d836c41152a76d4c06f79c980ca2d1127008281f63fd36daa08b65dea2e WatchSource:0}: Error finding container fd2b6d836c41152a76d4c06f79c980ca2d1127008281f63fd36daa08b65dea2e: Status 404 returned error can't find the container with id fd2b6d836c41152a76d4c06f79c980ca2d1127008281f63fd36daa08b65dea2e
Oct 30 14:36:17.223568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:17.223183    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"]
Oct 30 14:36:17.683631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:17.683597    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq" event={"ID":"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d","Type":"ContainerStarted","Data":"c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907"}
Oct 30 14:36:17.683806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:17.683661    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq" event={"ID":"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d","Type":"ContainerStarted","Data":"fd2b6d836c41152a76d4c06f79c980ca2d1127008281f63fd36daa08b65dea2e"}
Oct 30 14:36:17.683806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:17.683797    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:17.739260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:17.739205    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq" podStartSLOduration=2.434428557 podStartE2EDuration="2.7391905s" totalImagesPullingTime="304.761943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:36:17.734451701 +0000 UTC m=+2842.766350640" watchObservedRunningTime="2026-10-30 14:36:17.7391905 +0000 UTC m=+2842.771089449"
Oct 30 14:36:23.697477 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:23.697445    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:36:24.572587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:24.568626    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:36:35.880984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:35.880950    4530 generic.go:356] "Generic (PLEG): container finished" podID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerID="13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d" exitCode=0
Oct 30 14:36:35.881411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:35.881023    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" event={"ID":"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7","Type":"ContainerDied","Data":"13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d"}
Oct 30 14:36:35.881455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:35.881439    4530 scope.go:122] "RemoveContainer" containerID="13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d"
Oct 30 14:36:35.883098 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:35.883062    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:36:35.883233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:35.883148    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:36:35.883233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:35.883199    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:36:36.912938 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:36.912900    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" event={"ID":"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7","Type":"ContainerStarted","Data":"0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80"}
Oct 30 14:36:37.412129 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:37.412092    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:36:37.413227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:37.413198    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 14:36:37.413346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:37.413275    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 14:36:38.412322 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:38.412291    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 14:36:38.412835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:38.412807    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 14:36:39.412857 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:39.412822    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 14:36:39.413277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:39.412882    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 14:36:40.413187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:40.413146    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 14:36:40.413663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:40.413206    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 14:36:40.910200 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:40.910161    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:36:41.368146 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:36:41.368109    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:36:41.413194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:41.413159    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 14:36:41.413622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:41.413211    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 14:36:42.417363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:42.417329    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:36:42.418572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:42.418527    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:36:44.042435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.042401    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"]
Oct 30 14:36:44.045942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.045910    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.067962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.067932    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"]
Oct 30 14:36:44.073552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.073511    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d3a829-serving-cert\""
Oct 30 14:36:44.097103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.097059    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-metrics-certs\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.097249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.097225    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.097334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.097321    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcdtp\" (UniqueName: \"kubernetes.io/projected/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-kube-api-access-xcdtp\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.097399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.097376    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.198358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.198327    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-metrics-certs\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.198626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.198603    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.198793 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.198701    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcdtp\" (UniqueName: \"kubernetes.io/projected/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-kube-api-access-xcdtp\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.198793 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.198785    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.199019 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:36:44.198992    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-serving-cert: secret "maas-api-e2e-ipp-a-d3a829-serving-cert" not found
Oct 30 14:36:44.199144 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:36:44.199068    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-maas-api-tls podName:1e5cbbcf-0058-44e5-8e80-2c4fb100bf70 nodeName:}" failed. No retries permitted until 2026-10-30 14:36:44.699045677 +0000 UTC m=+2869.730944597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-maas-api-tls") pod "maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62" (UID: "1e5cbbcf-0058-44e5-8e80-2c4fb100bf70") : secret "maas-api-e2e-ipp-a-d3a829-serving-cert" not found
Oct 30 14:36:44.199424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.199394    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.201183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.201162    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-metrics-certs\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.255900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.255825    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcdtp\" (UniqueName: \"kubernetes.io/projected/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-kube-api-access-xcdtp\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.705657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.705628    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.708071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.708041    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:44.959772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:44.959693    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:45.288381 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:36:45.288348    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5cbbcf_0058_44e5_8e80_2c4fb100bf70.slice/crio-63c9d321761c3fa44d356dc3e5f56b100e1ebd7cbffdab40574f4c735db7a28f WatchSource:0}: Error finding container 63c9d321761c3fa44d356dc3e5f56b100e1ebd7cbffdab40574f4c735db7a28f: Status 404 returned error can't find the container with id 63c9d321761c3fa44d356dc3e5f56b100e1ebd7cbffdab40574f4c735db7a28f
Oct 30 14:36:45.288740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:45.288472    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"]
Oct 30 14:36:45.988922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:45.988822    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62" event={"ID":"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70","Type":"ContainerStarted","Data":"261b440cf2b1db5ac1c16ad38002a6fbb3197c6fa1e2fce0c45b4bedd5e7b5ad"}
Oct 30 14:36:45.988922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:45.988895    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62" event={"ID":"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70","Type":"ContainerStarted","Data":"63c9d321761c3fa44d356dc3e5f56b100e1ebd7cbffdab40574f4c735db7a28f"}
Oct 30 14:36:45.988922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:45.988924    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:46.046709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:46.046659    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62" podStartSLOduration=1.696659789 podStartE2EDuration="2.04663554s" totalImagesPullingTime="349.975751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:36:46.040063626 +0000 UTC m=+2871.071962568" watchObservedRunningTime="2026-10-30 14:36:46.04663554 +0000 UTC m=+2871.078534520"
Oct 30 14:36:52.001820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:52.001788    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:36:52.290854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:36:52.290778    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"]
Oct 30 14:37:02.568459 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:02.568425    4530 scope.go:122] "RemoveContainer" containerID="511741a7ce1ddda293b774d1c88438bac1a07b5248112bdfff08936a6ec595a0"
Oct 30 14:37:04.891955 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.891918    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"]
Oct 30 14:37:04.897195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.897169    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:04.927030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.926997    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"]
Oct 30 14:37:04.930830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.930802    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e50e63-openshift-default-dockercfg-n5qtw\""
Oct 30 14:37:04.975884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.975856    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43206177-2c02-4a0f-87da-a00340846626-istio-podinfo\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:04.976018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.975892    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43206177-2c02-4a0f-87da-a00340846626-istiod-ca-cert\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:04.976018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.975911    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm8dw\" (UniqueName: \"kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-kube-api-access-dm8dw\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:04.976126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.976031    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-certs\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:04.976126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.976082    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-data\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:04.976126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.976100    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-envoy\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:04.976287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.976137    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-istio-token\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:04.976287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.976237    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-socket\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:04.976492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:04.976308    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-credential-socket\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.090211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.090162    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-istio-token\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.090872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.090840    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-socket\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.091499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.091479    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-credential-socket\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.092149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.092129    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43206177-2c02-4a0f-87da-a00340846626-istio-podinfo\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.092303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.092285    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43206177-2c02-4a0f-87da-a00340846626-istiod-ca-cert\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.092417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.092402    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm8dw\" (UniqueName: \"kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-kube-api-access-dm8dw\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.092582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.092565    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-certs\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.092739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.092721    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-data\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.092861 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.092844    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-envoy\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.094045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.094023    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43206177-2c02-4a0f-87da-a00340846626-istiod-ca-cert\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.094932 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.091413    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-socket\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.095066 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.094431    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-certs\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.095162 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.091901    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-credential-socket\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.095249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.094829    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-data\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.130953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.130917    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-envoy\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.144534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.142947    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43206177-2c02-4a0f-87da-a00340846626-istio-podinfo\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.144534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.143396    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-istio-token\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.147745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.147715    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm8dw\" (UniqueName: \"kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-kube-api-access-dm8dw\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.218616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.218576    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:05.586437 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:37:05.586389    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43206177_2c02_4a0f_87da_a00340846626.slice/crio-ee00be75870f6d868d35dc10312d5575cc8f9642fcc30c3e12b0ae181e211e1d WatchSource:0}: Error finding container ee00be75870f6d868d35dc10312d5575cc8f9642fcc30c3e12b0ae181e211e1d: Status 404 returned error can't find the container with id ee00be75870f6d868d35dc10312d5575cc8f9642fcc30c3e12b0ae181e211e1d
Oct 30 14:37:05.593024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.592974    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:37:05.593159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.593125    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:37:05.593266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.593223    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:37:05.621188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:05.621136    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"]
Oct 30 14:37:06.159264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:06.159225    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" event={"ID":"43206177-2c02-4a0f-87da-a00340846626","Type":"ContainerStarted","Data":"250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d"}
Oct 30 14:37:06.159778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:06.159282    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" event={"ID":"43206177-2c02-4a0f-87da-a00340846626","Type":"ContainerStarted","Data":"ee00be75870f6d868d35dc10312d5575cc8f9642fcc30c3e12b0ae181e211e1d"}
Oct 30 14:37:06.217590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:06.217529    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" podStartSLOduration=2.217513994 podStartE2EDuration="2.217513994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:37:06.21334569 +0000 UTC m=+2891.245244629" watchObservedRunningTime="2026-10-30 14:37:06.217513994 +0000 UTC m=+2891.249412933"
Oct 30 14:37:06.218946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:06.218922    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:06.225180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:06.225159    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:07.177567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:07.173955    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:07.177567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:07.175507    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:37:07.327353 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:07.327129    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh" podUID="eb89111d-7a71-4de1-994a-7470dc21d733" containerName="main" containerID="cri-o://46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e" gracePeriod=45
Oct 30 14:37:08.074565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.073729    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:37:08.241566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.241286    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb89111d-7a71-4de1-994a-7470dc21d733-kube-api-access-xb26w\" (UniqueName: \"kubernetes.io/projected/eb89111d-7a71-4de1-994a-7470dc21d733-kube-api-access-xb26w\") pod \"eb89111d-7a71-4de1-994a-7470dc21d733\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") "
Oct 30 14:37:08.241566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.241353    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-dshm\") pod \"eb89111d-7a71-4de1-994a-7470dc21d733\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") "
Oct 30 14:37:08.241566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.241396    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-tmp-dir\") pod \"eb89111d-7a71-4de1-994a-7470dc21d733\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") "
Oct 30 14:37:08.241566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.241535    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb89111d-7a71-4de1-994a-7470dc21d733-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb89111d-7a71-4de1-994a-7470dc21d733-tls-certs\") pod \"eb89111d-7a71-4de1-994a-7470dc21d733\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") "
Oct 30 14:37:08.242207 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.241636    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-model-cache\") pod \"eb89111d-7a71-4de1-994a-7470dc21d733\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") "
Oct 30 14:37:08.242207 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.241735    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-home\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-home\") pod \"eb89111d-7a71-4de1-994a-7470dc21d733\" (UID: \"eb89111d-7a71-4de1-994a-7470dc21d733\") "
Oct 30 14:37:08.255564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.246635    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-home" pod "eb89111d-7a71-4de1-994a-7470dc21d733" (UID: "eb89111d-7a71-4de1-994a-7470dc21d733"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:37:08.269737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.269702    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-tmp-dir" pod "eb89111d-7a71-4de1-994a-7470dc21d733" (UID: "eb89111d-7a71-4de1-994a-7470dc21d733"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:37:08.274007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.271445    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-model-cache" pod "eb89111d-7a71-4de1-994a-7470dc21d733" (UID: "eb89111d-7a71-4de1-994a-7470dc21d733"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:37:08.275253 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.275149    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb89111d-7a71-4de1-994a-7470dc21d733-tls-certs" pod "eb89111d-7a71-4de1-994a-7470dc21d733" (UID: "eb89111d-7a71-4de1-994a-7470dc21d733"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:37:08.275253 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.275175    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb89111d-7a71-4de1-994a-7470dc21d733-kube-api-access-xb26w" pod "eb89111d-7a71-4de1-994a-7470dc21d733" (UID: "eb89111d-7a71-4de1-994a-7470dc21d733"). InnerVolumeSpecName "kube-api-access-xb26w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:37:08.329618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.323343    4530 generic.go:356] "Generic (PLEG): container finished" podID="eb89111d-7a71-4de1-994a-7470dc21d733" containerID="46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e" exitCode=0
Oct 30 14:37:08.329618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.325280    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"
Oct 30 14:37:08.329618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.325928    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh" event={"ID":"eb89111d-7a71-4de1-994a-7470dc21d733","Type":"ContainerDied","Data":"46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e"}
Oct 30 14:37:08.329618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.325991    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh" event={"ID":"eb89111d-7a71-4de1-994a-7470dc21d733","Type":"ContainerDied","Data":"ad09283ca6ee5d2b99bf32d3e476f997d63a0d0e19d543b2fe0045a8a9479458"}
Oct 30 14:37:08.329618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.326015    4530 scope.go:122] "RemoveContainer" containerID="46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e"
Oct 30 14:37:08.364103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.364058    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:08.364103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.364095    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:08.364103 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.364109    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb26w\" (UniqueName: \"kubernetes.io/projected/eb89111d-7a71-4de1-994a-7470dc21d733-kube-api-access-xb26w\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:08.364411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.364124    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:08.364411 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.364138    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb89111d-7a71-4de1-994a-7470dc21d733-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:08.409564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.409120    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-dshm" pod "eb89111d-7a71-4de1-994a-7470dc21d733" (UID: "eb89111d-7a71-4de1-994a-7470dc21d733"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:37:08.466487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.466435    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb89111d-7a71-4de1-994a-7470dc21d733-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:08.533034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.532997    4530 scope.go:122] "RemoveContainer" containerID="46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e"
Oct 30 14:37:08.533903 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:37:08.533877    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e\": container with ID starting with 46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e not found: ID does not exist" containerID="46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e"
Oct 30 14:37:08.534095 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.534063    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e"} err="failed to get container status \"46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e\": rpc error: code = NotFound desc = could not find container \"46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e\": container with ID starting with 46b83b653b80ff78f9cfbba8478beb8682145ed195f2900dd29694b082bf785e not found: ID does not exist"
Oct 30 14:37:08.745587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.744585    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"]
Oct 30 14:37:08.749867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:08.749816    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-75443e/auth-test-model-75443e-kserve-6777b67757-85xmh"]
Oct 30 14:37:09.140574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:09.139086    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb89111d-7a71-4de1-994a-7470dc21d733" path="/var/lib/kubelet/pods/eb89111d-7a71-4de1-994a-7470dc21d733/volumes"
Oct 30 14:37:11.370032 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:37:11.369990    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:37:17.410960 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:17.410927    4530 generic.go:356] "Generic (PLEG): container finished" podID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerID="184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472" exitCode=0
Oct 30 14:37:17.411401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:17.411013    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" event={"ID":"b2abb44c-4a6d-44b2-8081-d24b067dda00","Type":"ContainerDied","Data":"184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472"}
Oct 30 14:37:17.411462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:17.411423    4530 scope.go:122] "RemoveContainer" containerID="184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472"
Oct 30 14:37:17.413042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:17.413013    4530 generic.go:356] "Generic (PLEG): container finished" podID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerID="631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0" exitCode=0
Oct 30 14:37:17.413148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:17.413092    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" event={"ID":"330c4039-1ce9-4eee-af89-a18cbc37e9a0","Type":"ContainerDied","Data":"631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0"}
Oct 30 14:37:17.413286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:17.413254    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:37:17.413368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:17.413349    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:37:17.413417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:17.413400    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:37:17.413500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:17.413485    4530 scope.go:122] "RemoveContainer" containerID="631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0"
Oct 30 14:37:17.415093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:17.415066    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:37:17.415172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:17.415127    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:37:17.415172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:17.415157    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:37:18.437855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:18.437813    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" event={"ID":"b2abb44c-4a6d-44b2-8081-d24b067dda00","Type":"ContainerStarted","Data":"0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68"}
Oct 30 14:37:18.440155 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:18.440126    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" event={"ID":"330c4039-1ce9-4eee-af89-a18cbc37e9a0","Type":"ContainerStarted","Data":"9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093"}
Oct 30 14:37:18.463883 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:18.463860    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:37:18.465336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:18.465313    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 14:37:18.465440 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:18.465357    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 14:37:18.979347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:18.979302    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:37:18.992574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:18.991666    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 14:37:18.992574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:18.991736    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 14:37:19.334002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:19.333958    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"]
Oct 30 14:37:19.471566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:19.468291    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 14:37:19.471566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:19.468352    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 14:37:20.002566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:19.983656    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 14:37:20.002566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:19.983721    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 14:37:20.464161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.464099    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 14:37:20.464161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.464149    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 14:37:20.639267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.637451    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"]
Oct 30 14:37:20.639267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.638485    4530 cpu_manager.go:414] "containerMap: removing container" podUID="eb89111d-7a71-4de1-994a-7470dc21d733" containerName="main"
Oct 30 14:37:20.639267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.638509    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb89111d-7a71-4de1-994a-7470dc21d733" containerName="main"
Oct 30 14:37:20.639267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.638762    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb89111d-7a71-4de1-994a-7470dc21d733" containerName="main"
Oct 30 14:37:20.645588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.645564    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.647862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.647833    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"]
Oct 30 14:37:20.673110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.673079    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbbpv\" (UniqueName: \"kubernetes.io/projected/a389d6ca-7846-418f-83b7-82156f64fd88-kube-api-access-hbbpv\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.673244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.673218    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a389d6ca-7846-418f-83b7-82156f64fd88-tls-certs\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.673323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.673293    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-home\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.673387 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.673322    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-model-cache\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.673387 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.673366    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-tmp-dir\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.673508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.673480    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-dshm\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.678146 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.678119    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-9e3aa0\"/\"auth-test-model-9e3aa0-kserve-self-signed-certs\""
Oct 30 14:37:20.678268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.678174    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-9e3aa0\"/\"openshift-service-ca.crt\""
Oct 30 14:37:20.678268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.678197    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-9e3aa0\"/\"default-dockercfg-7qxwt\""
Oct 30 14:37:20.678402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.678285    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-9e3aa0\"/\"kube-root-ca.crt\""
Oct 30 14:37:20.774577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.774471    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a389d6ca-7846-418f-83b7-82156f64fd88-tls-certs\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.774577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.774532    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-home\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.774801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.774584    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-model-cache\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.774801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.774618    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-tmp-dir\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.774801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.774728    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-dshm\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.774981 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.774831    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbbpv\" (UniqueName: \"kubernetes.io/projected/a389d6ca-7846-418f-83b7-82156f64fd88-kube-api-access-hbbpv\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.775069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.775044    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-home\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.775141 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.775064    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-model-cache\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.775141 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.775124    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-tmp-dir\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.777061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.777019    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-dshm\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.777341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.777226    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a389d6ca-7846-418f-83b7-82156f64fd88-tls-certs\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.831327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.831258    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbbpv\" (UniqueName: \"kubernetes.io/projected/a389d6ca-7846-418f-83b7-82156f64fd88-kube-api-access-hbbpv\") pod \"auth-test-model-9e3aa0-kserve-89b45d65-clbb8\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") " pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:20.974222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.974185    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:21.012215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.980428    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 14:37:21.012215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:20.980492    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 14:37:21.135708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:21.135673    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:37:21.435005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:21.433939    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:37:21.437089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:21.436751    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"]
Oct 30 14:37:21.463805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:21.463777    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 14:37:21.463888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:21.463829    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 14:37:21.542758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:21.542724    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8" event={"ID":"a389d6ca-7846-418f-83b7-82156f64fd88","Type":"ContainerStarted","Data":"3d1bc742ed86602d26a90263d230fe9e02a20e968a2acc189ce811b183000955"}
Oct 30 14:37:21.979033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:21.979004    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 14:37:21.979474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:21.979451    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 14:37:22.464064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:22.464028    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 14:37:22.464238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:22.464079    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 14:37:22.702502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:22.701408    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8" event={"ID":"a389d6ca-7846-418f-83b7-82156f64fd88","Type":"ContainerStarted","Data":"cf8eb1529d82d8c4911cde4210956969114f8c00d6938f0a218bd7db6d1b4d93"}
Oct 30 14:37:22.795259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:22.793898    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8" podStartSLOduration=2.793875655 podStartE2EDuration="2.793875655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:37:22.758959987 +0000 UTC m=+2907.790858926" watchObservedRunningTime="2026-10-30 14:37:22.793875655 +0000 UTC m=+2907.825774595"
Oct 30 14:37:22.975374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:22.975333    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:22.979510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:22.979480    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 14:37:23.002956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:22.988609    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 14:37:23.464518 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:23.464482    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 14:37:23.464720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:23.464577    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 14:37:23.979063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:23.979028    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 14:37:23.979256 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:23.979085    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 14:37:24.468339 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:24.468308    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:37:24.469625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:24.469596    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:37:24.983994 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:24.983963    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:37:24.984985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:24.984961    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:37:30.975381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:30.975348    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:30.988090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:30.988065    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:31.448906 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:31.448873    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"]
Oct 30 14:37:31.832840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:31.832809    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:37:34.355799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.355751    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p" podUID="3db87e42-382c-4022-8a55-574b4edc1344" containerName="main" containerID="cri-o://c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191" gracePeriod=45
Oct 30 14:37:34.477304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.477269    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p" podUID="3db87e42-382c-4022-8a55-574b4edc1344" containerName="main" probeResult="failure" output="Get \"https://10.133.1.5:8000/ready\": dial tcp 10.133.1.5:8000: connect: connection refused"
Oct 30 14:37:34.632015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.631984    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:37:34.738830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.738801    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-model-cache\") pod \"3db87e42-382c-4022-8a55-574b4edc1344\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") "
Oct 30 14:37:34.738992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.738848    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-dshm\") pod \"3db87e42-382c-4022-8a55-574b4edc1344\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") "
Oct 30 14:37:34.738992 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.738955    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-home\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-home\") pod \"3db87e42-382c-4022-8a55-574b4edc1344\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") "
Oct 30 14:37:34.739066 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.738994    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-tmp-dir\") pod \"3db87e42-382c-4022-8a55-574b4edc1344\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") "
Oct 30 14:37:34.739066 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.739047    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3db87e42-382c-4022-8a55-574b4edc1344-tls-certs\" (UniqueName: \"kubernetes.io/secret/3db87e42-382c-4022-8a55-574b4edc1344-tls-certs\") pod \"3db87e42-382c-4022-8a55-574b4edc1344\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") "
Oct 30 14:37:34.739161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.739077    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db87e42-382c-4022-8a55-574b4edc1344-kube-api-access-wtpmz\" (UniqueName: \"kubernetes.io/projected/3db87e42-382c-4022-8a55-574b4edc1344-kube-api-access-wtpmz\") pod \"3db87e42-382c-4022-8a55-574b4edc1344\" (UID: \"3db87e42-382c-4022-8a55-574b4edc1344\") "
Oct 30 14:37:34.739161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.739081    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-model-cache" pod "3db87e42-382c-4022-8a55-574b4edc1344" (UID: "3db87e42-382c-4022-8a55-574b4edc1344"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:37:34.739277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.739219    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-home" pod "3db87e42-382c-4022-8a55-574b4edc1344" (UID: "3db87e42-382c-4022-8a55-574b4edc1344"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:37:34.739358 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.739332    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-tmp-dir" pod "3db87e42-382c-4022-8a55-574b4edc1344" (UID: "3db87e42-382c-4022-8a55-574b4edc1344"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:37:34.739676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.739649    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:34.739676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.739675    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:34.739886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.739689    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:34.741394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.741359    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3db87e42-382c-4022-8a55-574b4edc1344-tls-certs" pod "3db87e42-382c-4022-8a55-574b4edc1344" (UID: "3db87e42-382c-4022-8a55-574b4edc1344"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:37:34.741394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.741381    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db87e42-382c-4022-8a55-574b4edc1344-kube-api-access-wtpmz" pod "3db87e42-382c-4022-8a55-574b4edc1344" (UID: "3db87e42-382c-4022-8a55-574b4edc1344"). InnerVolumeSpecName "kube-api-access-wtpmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:37:34.741516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.741383    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-dshm" pod "3db87e42-382c-4022-8a55-574b4edc1344" (UID: "3db87e42-382c-4022-8a55-574b4edc1344"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:37:34.841126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.841092    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3db87e42-382c-4022-8a55-574b4edc1344-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:34.841126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.841123    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3db87e42-382c-4022-8a55-574b4edc1344-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:34.841360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.841141    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtpmz\" (UniqueName: \"kubernetes.io/projected/3db87e42-382c-4022-8a55-574b4edc1344-kube-api-access-wtpmz\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:34.846023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.845993    4530 generic.go:356] "Generic (PLEG): container finished" podID="3db87e42-382c-4022-8a55-574b4edc1344" containerID="c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191" exitCode=0
Oct 30 14:37:34.846167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.846032    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"
Oct 30 14:37:34.846167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.846084    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p" event={"ID":"3db87e42-382c-4022-8a55-574b4edc1344","Type":"ContainerDied","Data":"c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191"}
Oct 30 14:37:34.846167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.846138    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p" event={"ID":"3db87e42-382c-4022-8a55-574b4edc1344","Type":"ContainerDied","Data":"4ade57ced1e40ef57a335404fd68588b34b8d787c803648a6513e345c568c901"}
Oct 30 14:37:34.846167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.846165    4530 scope.go:122] "RemoveContainer" containerID="c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191"
Oct 30 14:37:34.859156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.859137    4530 scope.go:122] "RemoveContainer" containerID="c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191"
Oct 30 14:37:34.859455 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:37:34.859436    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191\": container with ID starting with c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191 not found: ID does not exist" containerID="c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191"
Oct 30 14:37:34.859531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.859465    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191"} err="failed to get container status \"c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191\": rpc error: code = NotFound desc = could not find container \"c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191\": container with ID starting with c66e67ba1e73cbcbfa24ca2fb4ba107339b89dea83ee8eeef250869dc8ec4191 not found: ID does not exist"
Oct 30 14:37:34.909144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.909083    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"]
Oct 30 14:37:34.914926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:34.914903    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0efe43/auth-test-model-0efe43-kserve-64b47fdcc-lch7p"]
Oct 30 14:37:35.121321 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:35.121287    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db87e42-382c-4022-8a55-574b4edc1344" path="/var/lib/kubelet/pods/3db87e42-382c-4022-8a55-574b4edc1344/volumes"
Oct 30 14:37:41.367826 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:37:41.367750    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:37:46.472414 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.472362    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd" podUID="23ff4fb9-73ff-4d09-a9a4-125a23a52995" containerName="main" containerID="cri-o://6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce" gracePeriod=45
Oct 30 14:37:46.745741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.745714    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:37:46.794346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.794310    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-home\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-home\") pod \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") "
Oct 30 14:37:46.794508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.794371    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tmp-dir\") pod \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") "
Oct 30 14:37:46.794508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.794424    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-model-cache\") pod \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") "
Oct 30 14:37:46.794508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.794486    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ff4fb9-73ff-4d09-a9a4-125a23a52995-kube-api-access-pq4sr\" (UniqueName: \"kubernetes.io/projected/23ff4fb9-73ff-4d09-a9a4-125a23a52995-kube-api-access-pq4sr\") pod \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") "
Oct 30 14:37:46.794759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.794618    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-home" pod "23ff4fb9-73ff-4d09-a9a4-125a23a52995" (UID: "23ff4fb9-73ff-4d09-a9a4-125a23a52995"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:37:46.794759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.794654    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tmp-dir" pod "23ff4fb9-73ff-4d09-a9a4-125a23a52995" (UID: "23ff4fb9-73ff-4d09-a9a4-125a23a52995"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:37:46.794759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.794697    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tls-certs\" (UniqueName: \"kubernetes.io/secret/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tls-certs\") pod \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") "
Oct 30 14:37:46.794759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.794746    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-dshm\") pod \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\" (UID: \"23ff4fb9-73ff-4d09-a9a4-125a23a52995\") "
Oct 30 14:37:46.794978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.794792    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-model-cache" pod "23ff4fb9-73ff-4d09-a9a4-125a23a52995" (UID: "23ff4fb9-73ff-4d09-a9a4-125a23a52995"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:37:46.795374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.795354    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:46.795435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.795380    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:46.795435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.795394    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:46.802760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.802710    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-dshm" pod "23ff4fb9-73ff-4d09-a9a4-125a23a52995" (UID: "23ff4fb9-73ff-4d09-a9a4-125a23a52995"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:37:46.802947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.802922    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ff4fb9-73ff-4d09-a9a4-125a23a52995-kube-api-access-pq4sr" pod "23ff4fb9-73ff-4d09-a9a4-125a23a52995" (UID: "23ff4fb9-73ff-4d09-a9a4-125a23a52995"). InnerVolumeSpecName "kube-api-access-pq4sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:37:46.804321 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.804291    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tls-certs" pod "23ff4fb9-73ff-4d09-a9a4-125a23a52995" (UID: "23ff4fb9-73ff-4d09-a9a4-125a23a52995"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:37:46.897227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.897199    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq4sr\" (UniqueName: \"kubernetes.io/projected/23ff4fb9-73ff-4d09-a9a4-125a23a52995-kube-api-access-pq4sr\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:46.897227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.897227    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23ff4fb9-73ff-4d09-a9a4-125a23a52995-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:46.897388 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.897236    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ff4fb9-73ff-4d09-a9a4-125a23a52995-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:37:46.972824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.972792    4530 generic.go:356] "Generic (PLEG): container finished" podID="23ff4fb9-73ff-4d09-a9a4-125a23a52995" containerID="6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce" exitCode=0
Oct 30 14:37:46.972988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.972840    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"
Oct 30 14:37:46.972988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.972858    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd" event={"ID":"23ff4fb9-73ff-4d09-a9a4-125a23a52995","Type":"ContainerDied","Data":"6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce"}
Oct 30 14:37:46.972988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.972909    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd" event={"ID":"23ff4fb9-73ff-4d09-a9a4-125a23a52995","Type":"ContainerDied","Data":"9cb6a82290c975ecbd5c152abd3181828bb32aad32d6e35815a78f131ce47ed1"}
Oct 30 14:37:46.972988 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.972925    4530 scope.go:122] "RemoveContainer" containerID="6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce"
Oct 30 14:37:46.983875 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.983854    4530 scope.go:122] "RemoveContainer" containerID="6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce"
Oct 30 14:37:46.984117 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:37:46.984098    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce\": container with ID starting with 6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce not found: ID does not exist" containerID="6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce"
Oct 30 14:37:46.984167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:46.984124    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce"} err="failed to get container status \"6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce\": rpc error: code = NotFound desc = could not find container \"6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce\": container with ID starting with 6afe5cff95298e104f76a53380e348971b1cd94d89a35be20c0c766b17f7e2ce not found: ID does not exist"
Oct 30 14:37:47.035582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:47.035527    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"]
Oct 30 14:37:47.042633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:47.042155    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5704d6/auth-test-model-5704d6-kserve-6cdd8c4949-bsknd"]
Oct 30 14:37:47.117123 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:47.117087    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ff4fb9-73ff-4d09-a9a4-125a23a52995" path="/var/lib/kubelet/pods/23ff4fb9-73ff-4d09-a9a4-125a23a52995/volumes"
Oct 30 14:37:51.614481 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:37:51.614442    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:38:02.371511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:02.371472    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"]
Oct 30 14:38:03.192674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.192637    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"]
Oct 30 14:38:03.193285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.193269    4530 cpu_manager.go:414] "containerMap: removing container" podUID="3db87e42-382c-4022-8a55-574b4edc1344" containerName="main"
Oct 30 14:38:03.193345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.193286    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db87e42-382c-4022-8a55-574b4edc1344" containerName="main"
Oct 30 14:38:03.193345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.193332    4530 cpu_manager.go:414] "containerMap: removing container" podUID="23ff4fb9-73ff-4d09-a9a4-125a23a52995" containerName="main"
Oct 30 14:38:03.193345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.193337    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ff4fb9-73ff-4d09-a9a4-125a23a52995" containerName="main"
Oct 30 14:38:03.193447 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.193438    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ff4fb9-73ff-4d09-a9a4-125a23a52995" containerName="main"
Oct 30 14:38:03.193484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.193448    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db87e42-382c-4022-8a55-574b4edc1344" containerName="main"
Oct 30 14:38:03.196974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.196957    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.215092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.215066    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"]
Oct 30 14:38:03.225039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.225017    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-e50e63-serving-cert\""
Oct 30 14:38:03.308553 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.308510    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4zqm\" (UniqueName: \"kubernetes.io/projected/e1f64945-8288-4cff-8760-118acf8e4427-kube-api-access-c4zqm\") pod \"maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.308738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.308612    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.308815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.308749    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f64945-8288-4cff-8760-118acf8e4427-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.308924 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.308897    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.410399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.410357    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f64945-8288-4cff-8760-118acf8e4427-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.411243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.410473    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.411243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.410719    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4zqm\" (UniqueName: \"kubernetes.io/projected/e1f64945-8288-4cff-8760-118acf8e4427-kube-api-access-c4zqm\") pod \"maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.411243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.410768    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.412140 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.412109    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f64945-8288-4cff-8760-118acf8e4427-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.419252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.419222    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"]
Oct 30 14:38:03.420390 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.420362    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.423436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.423405    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.425979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.425956    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.441218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.441192    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4zqm\" (UniqueName: \"kubernetes.io/projected/e1f64945-8288-4cff-8760-118acf8e4427-kube-api-access-c4zqm\") pod \"maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.442073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.442049    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"]
Oct 30 14:38:03.445155 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.445056    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f0e8b9-openshift-default-dockercfg-fhs75\""
Oct 30 14:38:03.521260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.515002    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:03.543568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.539567    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istiod-ca-cert\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.543568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.539677    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-data\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.543568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.539797    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-certs\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.543568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.539849    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-token\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.543568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.539881    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-credential-socket\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.543568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.539943    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-podinfo\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.543568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.539984    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-socket\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.543568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.540009    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp64x\" (UniqueName: \"kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-kube-api-access-fp64x\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.543568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.540034    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-envoy\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.641142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.641100    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-socket\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.641370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.641158    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp64x\" (UniqueName: \"kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-kube-api-access-fp64x\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.641370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.641184    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-envoy\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.641370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.641235    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istiod-ca-cert\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.641370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.641346    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-data\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.641673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.641495    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-certs\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.641673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.641576    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-token\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.641673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.641612    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-credential-socket\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.641828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.641683    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-podinfo\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.643117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.643084    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istiod-ca-cert\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.643418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.643391    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-socket\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.644253 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.644228    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-certs\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.644522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.644495    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-data\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.644826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.644805    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-credential-socket\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.647757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.647705    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-podinfo\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.648975 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.648950    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-envoy\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.698116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.698001    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-token\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.698313 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.698286    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp64x\" (UniqueName: \"kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-kube-api-access-fp64x\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.783387 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.783336    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:03.988733 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:38:03.988678    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f64945_8288_4cff_8760_118acf8e4427.slice/crio-f97b2578ac1fa0404d34593de51a8683d695f17142909af009db954805488464 WatchSource:0}: Error finding container f97b2578ac1fa0404d34593de51a8683d695f17142909af009db954805488464: Status 404 returned error can't find the container with id f97b2578ac1fa0404d34593de51a8683d695f17142909af009db954805488464
Oct 30 14:38:03.991041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:03.991017    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"]
Oct 30 14:38:04.113220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:04.113181    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"]
Oct 30 14:38:04.117102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:04.117009    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:38:04.117350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:04.117297    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:38:04.117467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:04.117361    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:38:04.131324 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:04.131296    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" event={"ID":"6a06488b-fc85-45f1-bbf9-a40a39125e5c","Type":"ContainerStarted","Data":"0038fc32c4a8a8b1a6e374dc3332c35d002e30f53043112c0f09ee5a82c7cfcf"}
Oct 30 14:38:04.132834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:04.132813    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb" event={"ID":"e1f64945-8288-4cff-8760-118acf8e4427","Type":"ContainerStarted","Data":"f97b2578ac1fa0404d34593de51a8683d695f17142909af009db954805488464"}
Oct 30 14:38:05.141962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:05.141894    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb" event={"ID":"e1f64945-8288-4cff-8760-118acf8e4427","Type":"ContainerStarted","Data":"faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26"}
Oct 30 14:38:05.142593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:05.142567    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:05.143914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:05.143884    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" event={"ID":"6a06488b-fc85-45f1-bbf9-a40a39125e5c","Type":"ContainerStarted","Data":"dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612"}
Oct 30 14:38:05.192575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:05.192495    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb" podStartSLOduration=1.818754378 podStartE2EDuration="2.192482472s" totalImagesPullingTime="373.728094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:05.190309555 +0000 UTC m=+2950.222208493" watchObservedRunningTime="2026-10-30 14:38:05.192482472 +0000 UTC m=+2950.224381411"
Oct 30 14:38:05.240563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:05.240500    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" podStartSLOduration=2.240488325 podStartE2EDuration="2.240488325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:05.239692165 +0000 UTC m=+2950.271591104" watchObservedRunningTime="2026-10-30 14:38:05.240488325 +0000 UTC m=+2950.272387327"
Oct 30 14:38:05.784211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:05.784175    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:05.791222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:05.791192    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:06.153581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:06.153501    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:06.154383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:06.154363    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:38:11.158230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:11.158200    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:38:11.368946 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:38:11.368902    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:38:14.987110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:14.987075    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"]
Oct 30 14:38:14.991157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:14.991139    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"
Oct 30 14:38:14.995089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:14.995066    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c3cc86-1167-4863-8f71-2c66a873b96a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9\" (UID: \"84c3cc86-1167-4863-8f71-2c66a873b96a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"
Oct 30 14:38:14.995200 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:14.995112    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt98h\" (UniqueName: \"kubernetes.io/projected/84c3cc86-1167-4863-8f71-2c66a873b96a-kube-api-access-bt98h\") pod \"payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9\" (UID: \"84c3cc86-1167-4863-8f71-2c66a873b96a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"
Oct 30 14:38:15.011288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:15.011261    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"]
Oct 30 14:38:15.019892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:15.019863    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d3a829\""
Oct 30 14:38:15.020024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:15.019926    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d3a829-dockercfg-jd4tt\""
Oct 30 14:38:15.096452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:15.096420    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c3cc86-1167-4863-8f71-2c66a873b96a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9\" (UID: \"84c3cc86-1167-4863-8f71-2c66a873b96a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"
Oct 30 14:38:15.096620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:15.096463    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt98h\" (UniqueName: \"kubernetes.io/projected/84c3cc86-1167-4863-8f71-2c66a873b96a-kube-api-access-bt98h\") pod \"payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9\" (UID: \"84c3cc86-1167-4863-8f71-2c66a873b96a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"
Oct 30 14:38:15.097218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:15.097197    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c3cc86-1167-4863-8f71-2c66a873b96a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9\" (UID: \"84c3cc86-1167-4863-8f71-2c66a873b96a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"
Oct 30 14:38:15.123528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:15.123500    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt98h\" (UniqueName: \"kubernetes.io/projected/84c3cc86-1167-4863-8f71-2c66a873b96a-kube-api-access-bt98h\") pod \"payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9\" (UID: \"84c3cc86-1167-4863-8f71-2c66a873b96a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"
Oct 30 14:38:15.306115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:15.306084    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"
Oct 30 14:38:15.640788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:15.640186    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"]
Oct 30 14:38:16.236300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:16.236253    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9" event={"ID":"84c3cc86-1167-4863-8f71-2c66a873b96a","Type":"ContainerStarted","Data":"51ea6e95438135a54b2fe1bee82e8864c2e776177c1269438eb0b6082dae061c"}
Oct 30 14:38:17.245175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.245139    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9" event={"ID":"84c3cc86-1167-4863-8f71-2c66a873b96a","Type":"ContainerStarted","Data":"3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c"}
Oct 30 14:38:17.245567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.245207    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"
Oct 30 14:38:17.292536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.292480    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9" podStartSLOduration=1.852417745 podStartE2EDuration="3.292461405s" totalImagesPullingTime="1.44004366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:17.290776413 +0000 UTC m=+2962.322675352" watchObservedRunningTime="2026-10-30 14:38:17.292461405 +0000 UTC m=+2962.324360344"
Oct 30 14:38:17.393285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.393204    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv" podUID="779351d5-97eb-4384-af2a-293f1f3cbdca" containerName="main" containerID="cri-o://c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b" gracePeriod=45
Oct 30 14:38:17.759593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.755410    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:38:17.884571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.876785    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/779351d5-97eb-4384-af2a-293f1f3cbdca-tls-certs\" (UniqueName: \"kubernetes.io/secret/779351d5-97eb-4384-af2a-293f1f3cbdca-tls-certs\") pod \"779351d5-97eb-4384-af2a-293f1f3cbdca\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") "
Oct 30 14:38:17.884571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.876903    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-dshm\") pod \"779351d5-97eb-4384-af2a-293f1f3cbdca\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") "
Oct 30 14:38:17.884571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.876980    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/779351d5-97eb-4384-af2a-293f1f3cbdca-kube-api-access-tmvs7\" (UniqueName: \"kubernetes.io/projected/779351d5-97eb-4384-af2a-293f1f3cbdca-kube-api-access-tmvs7\") pod \"779351d5-97eb-4384-af2a-293f1f3cbdca\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") "
Oct 30 14:38:17.884571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.877069    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-tmp-dir\") pod \"779351d5-97eb-4384-af2a-293f1f3cbdca\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") "
Oct 30 14:38:17.884571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.877127    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-home\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-home\") pod \"779351d5-97eb-4384-af2a-293f1f3cbdca\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") "
Oct 30 14:38:17.884571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.877320    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-model-cache\") pod \"779351d5-97eb-4384-af2a-293f1f3cbdca\" (UID: \"779351d5-97eb-4384-af2a-293f1f3cbdca\") "
Oct 30 14:38:17.885883 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.885851    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-model-cache" pod "779351d5-97eb-4384-af2a-293f1f3cbdca" (UID: "779351d5-97eb-4384-af2a-293f1f3cbdca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:38:17.889869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.889519    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779351d5-97eb-4384-af2a-293f1f3cbdca-tls-certs" pod "779351d5-97eb-4384-af2a-293f1f3cbdca" (UID: "779351d5-97eb-4384-af2a-293f1f3cbdca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:38:17.915921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.903964    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-dshm" pod "779351d5-97eb-4384-af2a-293f1f3cbdca" (UID: "779351d5-97eb-4384-af2a-293f1f3cbdca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:38:17.915921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.908789    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-tmp-dir" pod "779351d5-97eb-4384-af2a-293f1f3cbdca" (UID: "779351d5-97eb-4384-af2a-293f1f3cbdca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:38:17.915921 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.908985    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-home" pod "779351d5-97eb-4384-af2a-293f1f3cbdca" (UID: "779351d5-97eb-4384-af2a-293f1f3cbdca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:38:17.938787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.938724    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779351d5-97eb-4384-af2a-293f1f3cbdca-kube-api-access-tmvs7" pod "779351d5-97eb-4384-af2a-293f1f3cbdca" (UID: "779351d5-97eb-4384-af2a-293f1f3cbdca"). InnerVolumeSpecName "kube-api-access-tmvs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:38:17.993826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.993739    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:38:17.993826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.993778    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779351d5-97eb-4384-af2a-293f1f3cbdca-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:38:17.993826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.993791    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:38:17.993826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.993805    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmvs7\" (UniqueName: \"kubernetes.io/projected/779351d5-97eb-4384-af2a-293f1f3cbdca-kube-api-access-tmvs7\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:38:17.993826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.993819    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:38:17.994209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:17.993835    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779351d5-97eb-4384-af2a-293f1f3cbdca-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:38:18.254377 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:18.254295    4530 generic.go:356] "Generic (PLEG): container finished" podID="779351d5-97eb-4384-af2a-293f1f3cbdca" containerID="c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b" exitCode=0
Oct 30 14:38:18.254377 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:18.254341    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"
Oct 30 14:38:18.254377 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:18.254365    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv" event={"ID":"779351d5-97eb-4384-af2a-293f1f3cbdca","Type":"ContainerDied","Data":"c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b"}
Oct 30 14:38:18.254962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:18.254407    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv" event={"ID":"779351d5-97eb-4384-af2a-293f1f3cbdca","Type":"ContainerDied","Data":"8e84bcc8f226dd70ad5a788825710c698c619ef2ffda0da673bcf0cd3a1547bc"}
Oct 30 14:38:18.254962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:18.254431    4530 scope.go:122] "RemoveContainer" containerID="c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b"
Oct 30 14:38:18.265587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:18.265566    4530 scope.go:122] "RemoveContainer" containerID="c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b"
Oct 30 14:38:18.265866 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:38:18.265843    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b\": container with ID starting with c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b not found: ID does not exist" containerID="c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b"
Oct 30 14:38:18.265940 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:18.265878    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b"} err="failed to get container status \"c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b\": rpc error: code = NotFound desc = could not find container \"c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b\": container with ID starting with c00ac42997356626bdca691252623fb9ca51bf9bbebcdea6612339c5c52ead0b not found: ID does not exist"
Oct 30 14:38:18.321006 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:18.320981    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"]
Oct 30 14:38:18.325828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:18.325801    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-228454/auth-test-model-228454-kserve-5c4989b757-dfjlv"]
Oct 30 14:38:19.117844 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:19.117809    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="779351d5-97eb-4384-af2a-293f1f3cbdca" path="/var/lib/kubelet/pods/779351d5-97eb-4384-af2a-293f1f3cbdca/volumes"
Oct 30 14:38:28.257153 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:28.257118    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"
Oct 30 14:38:41.367878 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:38:41.367841    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:38:49.514093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:49.514058    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9591-237161"]
Oct 30 14:38:49.514791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:49.514774    4530 cpu_manager.go:414] "containerMap: removing container" podUID="779351d5-97eb-4384-af2a-293f1f3cbdca" containerName="main"
Oct 30 14:38:49.514846 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:49.514792    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="779351d5-97eb-4384-af2a-293f1f3cbdca" containerName="main"
Oct 30 14:38:49.514942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:49.514932    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="779351d5-97eb-4384-af2a-293f1f3cbdca" containerName="main"
Oct 30 14:38:49.520934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:49.520910    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-237161"
Oct 30 14:38:49.546074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:49.546040    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9591-237161"]
Oct 30 14:38:49.546595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:49.546566    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q95l7\""
Oct 30 14:38:49.600106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:49.600075    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsts\" (UniqueName: \"kubernetes.io/projected/48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921-kube-api-access-jnsts\") pod \"test-curl-9591-237161\" (UID: \"48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921\") " pod="opendatahub/test-curl-9591-237161"
Oct 30 14:38:49.701161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:49.701133    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsts\" (UniqueName: \"kubernetes.io/projected/48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921-kube-api-access-jnsts\") pod \"test-curl-9591-237161\" (UID: \"48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921\") " pod="opendatahub/test-curl-9591-237161"
Oct 30 14:38:49.728622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:49.728596    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsts\" (UniqueName: \"kubernetes.io/projected/48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921-kube-api-access-jnsts\") pod \"test-curl-9591-237161\" (UID: \"48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921\") " pod="opendatahub/test-curl-9591-237161"
Oct 30 14:38:49.855269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:49.855244    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-237161"
Oct 30 14:38:50.197038 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:38:50.197007    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48c09ca3_ee9f_4a89_ad7f_9a9e43f0e921.slice/crio-385bd045f7a873691f195773aba7085ecc50e2e203f2e0254722bdcbb095e950 WatchSource:0}: Error finding container 385bd045f7a873691f195773aba7085ecc50e2e203f2e0254722bdcbb095e950: Status 404 returned error can't find the container with id 385bd045f7a873691f195773aba7085ecc50e2e203f2e0254722bdcbb095e950
Oct 30 14:38:50.200272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:50.200246    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9591-237161"]
Oct 30 14:38:50.515375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:50.515287    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9591-237161" event={"ID":"48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921","Type":"ContainerStarted","Data":"c7165dc1572454164b615f6493193aff14e5fbbd37d91bdd1737a725eef71e16"}
Oct 30 14:38:50.515375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:50.515338    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9591-237161" event={"ID":"48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921","Type":"ContainerStarted","Data":"385bd045f7a873691f195773aba7085ecc50e2e203f2e0254722bdcbb095e950"}
Oct 30 14:38:50.553393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:50.553356    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"]
Oct 30 14:38:50.559759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:50.559704    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9591-237161" podStartSLOduration=1.5596846599999998 podStartE2EDuration="1.55968466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:50.559275144 +0000 UTC m=+2995.591174084" watchObservedRunningTime="2026-10-30 14:38:50.55968466 +0000 UTC m=+2995.591583601"
Oct 30 14:38:51.489512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:51.489441    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9591-237161"]
Oct 30 14:38:51.495295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:51.495264    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9591-237161"]
Oct 30 14:38:52.537407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:52.537366    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9591-237161" podUID="48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921" containerName="test-curl-9591-237161" containerID="cri-o://c7165dc1572454164b615f6493193aff14e5fbbd37d91bdd1737a725eef71e16" gracePeriod=2
Oct 30 14:38:52.553822 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:38:52.553788    4530 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9591-237161\" is forbidden: User \"system:node:ip-10-0-133-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-118.ec2.internal' and this object" podUID="48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921" pod="opendatahub/test-curl-9591-237161"
Oct 30 14:38:53.334634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:53.334600    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:38:54.976724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:54.976695    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-237161"
Oct 30 14:38:55.039879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:55.039847    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921-kube-api-access-jnsts\" (UniqueName: \"kubernetes.io/projected/48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921-kube-api-access-jnsts\") pod \"48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921\" (UID: \"48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921\") "
Oct 30 14:38:55.042120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:55.042090    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921-kube-api-access-jnsts" pod "48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921" (UID: "48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921"). InnerVolumeSpecName "kube-api-access-jnsts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:38:55.117884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:55.117852    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921" path="/var/lib/kubelet/pods/48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921/volumes"
Oct 30 14:38:55.141227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:55.141200    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnsts\" (UniqueName: \"kubernetes.io/projected/48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921-kube-api-access-jnsts\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:38:55.564474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:55.564433    4530 generic.go:356] "Generic (PLEG): container finished" podID="48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921" containerID="c7165dc1572454164b615f6493193aff14e5fbbd37d91bdd1737a725eef71e16" exitCode=137
Oct 30 14:38:55.564474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:55.564472    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-237161"
Oct 30 14:38:55.564706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:55.564521    4530 scope.go:122] "RemoveContainer" containerID="c7165dc1572454164b615f6493193aff14e5fbbd37d91bdd1737a725eef71e16"
Oct 30 14:38:55.574804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:55.574784    4530 scope.go:122] "RemoveContainer" containerID="c7165dc1572454164b615f6493193aff14e5fbbd37d91bdd1737a725eef71e16"
Oct 30 14:38:55.575093 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:38:55.575071    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7165dc1572454164b615f6493193aff14e5fbbd37d91bdd1737a725eef71e16\": container with ID starting with c7165dc1572454164b615f6493193aff14e5fbbd37d91bdd1737a725eef71e16 not found: ID does not exist" containerID="c7165dc1572454164b615f6493193aff14e5fbbd37d91bdd1737a725eef71e16"
Oct 30 14:38:55.575141 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:55.575103    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7165dc1572454164b615f6493193aff14e5fbbd37d91bdd1737a725eef71e16"} err="failed to get container status \"c7165dc1572454164b615f6493193aff14e5fbbd37d91bdd1737a725eef71e16\": rpc error: code = NotFound desc = could not find container \"c7165dc1572454164b615f6493193aff14e5fbbd37d91bdd1737a725eef71e16\": container with ID starting with c7165dc1572454164b615f6493193aff14e5fbbd37d91bdd1737a725eef71e16 not found: ID does not exist"
Oct 30 14:38:56.030369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:56.030291    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:38:56.032177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:56.032154    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:38:56.035165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:56.035143    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:38:56.036055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:56.036031    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:38:56.036759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:56.036740    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:38:56.037651 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:38:56.037631    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:39:01.666097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:01.666067    4530 generic.go:356] "Generic (PLEG): container finished" podID="43206177-2c02-4a0f-87da-a00340846626" containerID="250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d" exitCode=0
Oct 30 14:39:01.666456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:01.666142    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" event={"ID":"43206177-2c02-4a0f-87da-a00340846626","Type":"ContainerDied","Data":"250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d"}
Oct 30 14:39:01.666622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:01.666606    4530 scope.go:122] "RemoveContainer" containerID="250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d"
Oct 30 14:39:01.668445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:01.668413    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:39:01.668527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:01.668487    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:39:01.668527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:01.668518    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:39:02.712637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:02.712598    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" event={"ID":"43206177-2c02-4a0f-87da-a00340846626","Type":"ContainerStarted","Data":"390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd"}
Oct 30 14:39:03.219272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:03.219233    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:39:03.220456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:03.220428    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 14:39:03.220613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:03.220490    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" podUID="43206177-2c02-4a0f-87da-a00340846626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 14:39:04.219890 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:04.219856    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 14:39:04.220293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:04.219922    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" podUID="43206177-2c02-4a0f-87da-a00340846626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 14:39:05.219892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:05.219853    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 14:39:05.220070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:05.219921    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" podUID="43206177-2c02-4a0f-87da-a00340846626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 14:39:05.577014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:05.576965    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8" podUID="a389d6ca-7846-418f-83b7-82156f64fd88" containerName="main" containerID="cri-o://cf8eb1529d82d8c4911cde4210956969114f8c00d6938f0a218bd7db6d1b4d93" gracePeriod=45
Oct 30 14:39:05.767554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:05.767513    4530 generic.go:356] "Generic (PLEG): container finished" podID="a389d6ca-7846-418f-83b7-82156f64fd88" containerID="cf8eb1529d82d8c4911cde4210956969114f8c00d6938f0a218bd7db6d1b4d93" exitCode=0
Oct 30 14:39:05.767828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:05.767813    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8" event={"ID":"a389d6ca-7846-418f-83b7-82156f64fd88","Type":"ContainerDied","Data":"cf8eb1529d82d8c4911cde4210956969114f8c00d6938f0a218bd7db6d1b4d93"}
Oct 30 14:39:05.820017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:05.819967    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8" podUID="a389d6ca-7846-418f-83b7-82156f64fd88" containerName="main" probeResult="failure" output="Get \"https://10.133.1.20:8000/ready\": dial tcp 10.133.1.20:8000: connect: connection refused"
Oct 30 14:39:06.014710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.014680    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:39:06.120576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.120452    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a389d6ca-7846-418f-83b7-82156f64fd88-tls-certs\" (UniqueName: \"kubernetes.io/secret/a389d6ca-7846-418f-83b7-82156f64fd88-tls-certs\") pod \"a389d6ca-7846-418f-83b7-82156f64fd88\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") "
Oct 30 14:39:06.120784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.120653    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-home\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-home\") pod \"a389d6ca-7846-418f-83b7-82156f64fd88\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") "
Oct 30 14:39:06.120784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.120715    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-model-cache\") pod \"a389d6ca-7846-418f-83b7-82156f64fd88\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") "
Oct 30 14:39:06.120942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.120827    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a389d6ca-7846-418f-83b7-82156f64fd88-kube-api-access-hbbpv\" (UniqueName: \"kubernetes.io/projected/a389d6ca-7846-418f-83b7-82156f64fd88-kube-api-access-hbbpv\") pod \"a389d6ca-7846-418f-83b7-82156f64fd88\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") "
Oct 30 14:39:06.120942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.120878    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-dshm\") pod \"a389d6ca-7846-418f-83b7-82156f64fd88\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") "
Oct 30 14:39:06.120942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.120914    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-tmp-dir\") pod \"a389d6ca-7846-418f-83b7-82156f64fd88\" (UID: \"a389d6ca-7846-418f-83b7-82156f64fd88\") "
Oct 30 14:39:06.121786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.121758    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-home" pod "a389d6ca-7846-418f-83b7-82156f64fd88" (UID: "a389d6ca-7846-418f-83b7-82156f64fd88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:39:06.122056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.121780    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-tmp-dir" pod "a389d6ca-7846-418f-83b7-82156f64fd88" (UID: "a389d6ca-7846-418f-83b7-82156f64fd88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:39:06.122267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.122246    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-model-cache" pod "a389d6ca-7846-418f-83b7-82156f64fd88" (UID: "a389d6ca-7846-418f-83b7-82156f64fd88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:39:06.124701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.124675    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a389d6ca-7846-418f-83b7-82156f64fd88-kube-api-access-hbbpv" pod "a389d6ca-7846-418f-83b7-82156f64fd88" (UID: "a389d6ca-7846-418f-83b7-82156f64fd88"). InnerVolumeSpecName "kube-api-access-hbbpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:39:06.126430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.126405    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a389d6ca-7846-418f-83b7-82156f64fd88-tls-certs" pod "a389d6ca-7846-418f-83b7-82156f64fd88" (UID: "a389d6ca-7846-418f-83b7-82156f64fd88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:39:06.128644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.128620    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-dshm" pod "a389d6ca-7846-418f-83b7-82156f64fd88" (UID: "a389d6ca-7846-418f-83b7-82156f64fd88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:39:06.219461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.219422    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 14:39:06.219668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.219489    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" podUID="43206177-2c02-4a0f-87da-a00340846626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 14:39:06.223359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.223320    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:39:06.223359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.223352    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbbpv\" (UniqueName: \"kubernetes.io/projected/a389d6ca-7846-418f-83b7-82156f64fd88-kube-api-access-hbbpv\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:39:06.223949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.223366    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:39:06.223949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.223402    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:39:06.223949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.223416    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a389d6ca-7846-418f-83b7-82156f64fd88-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:39:06.223949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.223428    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a389d6ca-7846-418f-83b7-82156f64fd88-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:39:06.777375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.777341    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"
Oct 30 14:39:06.777593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.777391    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8" event={"ID":"a389d6ca-7846-418f-83b7-82156f64fd88","Type":"ContainerDied","Data":"3d1bc742ed86602d26a90263d230fe9e02a20e968a2acc189ce811b183000955"}
Oct 30 14:39:06.777593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.777444    4530 scope.go:122] "RemoveContainer" containerID="cf8eb1529d82d8c4911cde4210956969114f8c00d6938f0a218bd7db6d1b4d93"
Oct 30 14:39:06.865099 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.865065    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"]
Oct 30 14:39:06.893564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:06.879071    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9e3aa0/auth-test-model-9e3aa0-kserve-89b45d65-clbb8"]
Oct 30 14:39:07.230781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.230744    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 14:39:07.232494 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.232456    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" podUID="43206177-2c02-4a0f-87da-a00340846626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 14:39:07.324480 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.324446    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a389d6ca-7846-418f-83b7-82156f64fd88" path="/var/lib/kubelet/pods/a389d6ca-7846-418f-83b7-82156f64fd88/volumes"
Oct 30 14:39:07.324889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.324872    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:39:07.710562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.710463    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"]
Oct 30 14:39:07.711194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.711179    4530 cpu_manager.go:414] "containerMap: removing container" podUID="a389d6ca-7846-418f-83b7-82156f64fd88" containerName="main"
Oct 30 14:39:07.711238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.711195    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a389d6ca-7846-418f-83b7-82156f64fd88" containerName="main"
Oct 30 14:39:07.711238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.711211    4530 cpu_manager.go:414] "containerMap: removing container" podUID="48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921" containerName="test-curl-9591-237161"
Oct 30 14:39:07.711238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.711216    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921" containerName="test-curl-9591-237161"
Oct 30 14:39:07.711337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.711318    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c09ca3-ee9f-4a89-ad7f-9a9e43f0e921" containerName="test-curl-9591-237161"
Oct 30 14:39:07.711370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.711339    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="a389d6ca-7846-418f-83b7-82156f64fd88" containerName="main"
Oct 30 14:39:07.714875 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.714858    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.732445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.732419    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"]
Oct 30 14:39:07.741657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.741632    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-0efe43-openshift-default-dockercfg-bthxx\""
Oct 30 14:39:07.757169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.757139    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8qc9\" (UniqueName: \"kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-kube-api-access-m8qc9\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.757298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.757187    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-podinfo\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.757298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.757218    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.757376 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.757301    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-data\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.757434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.757417    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-envoy\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.757482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.757467    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-token\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.757642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.757620    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istiod-ca-cert\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.757703 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.757687    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-certs\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.757850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.757816    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-credential-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.859696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.859656    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-token\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.859916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.859827    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istiod-ca-cert\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.860000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.859921    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-certs\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.860053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.860035    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-credential-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.860216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.860199    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8qc9\" (UniqueName: \"kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-kube-api-access-m8qc9\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.860269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.860240    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-podinfo\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.860332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.860268    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.860398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.860328    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-certs\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.860398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.860338    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-data\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.860398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.860373    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-credential-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.860575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.860457    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-envoy\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.860653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.860623    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istiod-ca-cert\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.860726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.860663    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.860726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.860688    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-data\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.862831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.862801    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-envoy\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.862943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.862892    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-podinfo\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.894993 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.894970    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-token\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:07.895847 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:07.895827    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8qc9\" (UniqueName: \"kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-kube-api-access-m8qc9\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:08.029270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:08.029177    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:08.225533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:08.224218    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:39:08.229015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:08.227364    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:39:08.467155 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:39:08.467115    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c92412_886d_4c69_9c8e_3cf6fa1a344c.slice/crio-a86f12315e045ae242cf0de160b3337a0fdfd77af36ea97534cb6a169f6c3cff WatchSource:0}: Error finding container a86f12315e045ae242cf0de160b3337a0fdfd77af36ea97534cb6a169f6c3cff: Status 404 returned error can't find the container with id a86f12315e045ae242cf0de160b3337a0fdfd77af36ea97534cb6a169f6c3cff
Oct 30 14:39:08.468887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:08.468854    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"]
Oct 30 14:39:08.470271 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:08.470236    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:39:08.470387 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:08.470301    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:39:08.470387 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:08.470338    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:39:08.807733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:08.807701    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" event={"ID":"f2c92412-886d-4c69-9c8e-3cf6fa1a344c","Type":"ContainerStarted","Data":"ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112"}
Oct 30 14:39:08.807916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:08.807763    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" event={"ID":"f2c92412-886d-4c69-9c8e-3cf6fa1a344c","Type":"ContainerStarted","Data":"a86f12315e045ae242cf0de160b3337a0fdfd77af36ea97534cb6a169f6c3cff"}
Oct 30 14:39:08.862514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:08.862457    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" podStartSLOduration=1.862396406 podStartE2EDuration="1.862396406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:08.858715401 +0000 UTC m=+3013.890614363" watchObservedRunningTime="2026-10-30 14:39:08.862396406 +0000 UTC m=+3013.894295346"
Oct 30 14:39:09.030086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:09.030055    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:09.031614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:09.031587    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 14:39:09.031764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:09.031639    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" podUID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 14:39:10.029573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:10.029502    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 14:39:10.030068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:10.029589    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" podUID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 14:39:11.030168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:11.030130    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 14:39:11.030685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:11.030180    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" podUID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 14:39:11.367861 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:39:11.367823    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:39:12.030478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:12.030442    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 14:39:12.030952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:12.030496    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" podUID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 14:39:13.029629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:13.029588    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 14:39:13.029790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:13.029648    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" podUID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 14:39:14.030010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:14.029977    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 14:39:14.030384 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:14.030032    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" podUID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 14:39:15.030197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:15.030155    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 14:39:15.030662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:15.030227    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" podUID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 14:39:16.034737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:16.034702    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:16.035147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:16.035118    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:16.035749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:16.035731    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:39:19.986820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:19.986779    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"]
Oct 30 14:39:19.991883 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:19.991857    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.016183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.016154    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"]
Oct 30 14:39:20.026754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.026728    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-c7c37b\"/\"openshift-service-ca.crt\""
Oct 30 14:39:20.028789 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.028740    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-c7c37b\"/\"auth-test-model-c7c37b-kserve-self-signed-certs\""
Oct 30 14:39:20.028789 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.028777    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-c7c37b\"/\"kube-root-ca.crt\""
Oct 30 14:39:20.029025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.028780    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-c7c37b\"/\"default-dockercfg-99qh8\""
Oct 30 14:39:20.075452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.075418    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hk5\" (UniqueName: \"kubernetes.io/projected/bcb56386-09e1-4fa0-9a80-f291fc293fd1-kube-api-access-w6hk5\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.075632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.075484    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-model-cache\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.075632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.075597    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-dshm\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.075793 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.075772    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-home\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.075832 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.075818    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tmp-dir\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.075869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.075838    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tls-certs\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.177575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.177512    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-home\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.177751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.177609    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tmp-dir\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.177751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.177633    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tls-certs\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.177751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.177739    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hk5\" (UniqueName: \"kubernetes.io/projected/bcb56386-09e1-4fa0-9a80-f291fc293fd1-kube-api-access-w6hk5\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.177919 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.177808    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-model-cache\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.177919 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.177837    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-dshm\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.178022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.177956    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-home\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.178022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.177976    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tmp-dir\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.178199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.178172    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-model-cache\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.180028 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.180010    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-dshm\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.180428 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.180411    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tls-certs\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.205334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.205306    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hk5\" (UniqueName: \"kubernetes.io/projected/bcb56386-09e1-4fa0-9a80-f291fc293fd1-kube-api-access-w6hk5\") pod \"auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") " pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.309870 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.309827    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:20.673322 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:39:20.673289    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcb56386_09e1_4fa0_9a80_f291fc293fd1.slice/crio-f84752bc013d3f60ec1daf9ac843951bd5544ab49aa7c2488d2fea67386f5fc1 WatchSource:0}: Error finding container f84752bc013d3f60ec1daf9ac843951bd5544ab49aa7c2488d2fea67386f5fc1: Status 404 returned error can't find the container with id f84752bc013d3f60ec1daf9ac843951bd5544ab49aa7c2488d2fea67386f5fc1
Oct 30 14:39:20.675234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.675211    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"]
Oct 30 14:39:20.915014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.914977    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2" event={"ID":"bcb56386-09e1-4fa0-9a80-f291fc293fd1","Type":"ContainerStarted","Data":"98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41"}
Oct 30 14:39:20.915014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.915018    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2" event={"ID":"bcb56386-09e1-4fa0-9a80-f291fc293fd1","Type":"ContainerStarted","Data":"f84752bc013d3f60ec1daf9ac843951bd5544ab49aa7c2488d2fea67386f5fc1"}
Oct 30 14:39:20.966720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:20.966609    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2" podStartSLOduration=1.9665925990000002 podStartE2EDuration="1.966592599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:20.96303535 +0000 UTC m=+3025.994934290" watchObservedRunningTime="2026-10-30 14:39:20.966592599 +0000 UTC m=+3025.998491538"
Oct 30 14:39:21.310646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:21.310611    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:22.509566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:22.507644    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" podUID="e329b8d309e1f8141c44f91d4bb26de8" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 14:39:30.316586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:30.312964    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:30.329772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:30.329742    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:31.041385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:31.041353    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:39:39.877401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:39.877365    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"]
Oct 30 14:39:41.367774 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:39:41.367738    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:39:44.083689 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.083652    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"]
Oct 30 14:39:44.089008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.088980    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.108086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.108063    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6176c3-openshift-default-dockercfg-8z2pr\""
Oct 30 14:39:44.132567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.132520    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"]
Oct 30 14:39:44.188137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.188102    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxjjb\" (UniqueName: \"kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-kube-api-access-sxjjb\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.188137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.188138    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-podinfo\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.188354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.188167    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-envoy\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.188354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.188269    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57850de-5a13-4f64-9e13-004cbd6eb898-istiod-ca-cert\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.188354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.188317    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-socket\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.188459 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.188358    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-token\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.188459 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.188417    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-certs\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.188459 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.188456    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-data\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.188622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.188606    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-credential-socket\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.291573    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-envoy\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.291662    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57850de-5a13-4f64-9e13-004cbd6eb898-istiod-ca-cert\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.291714    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-socket\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.291740    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-token\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.291794    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-certs\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.291851    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-data\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.291957    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-credential-socket\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.292141    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxjjb\" (UniqueName: \"kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-kube-api-access-sxjjb\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.292173    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-podinfo\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.301366    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-data\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.301671    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-certs\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.302016    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-credential-socket\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.302683    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57850de-5a13-4f64-9e13-004cbd6eb898-istiod-ca-cert\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.313567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.302949    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-socket\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.325559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.321012    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-envoy\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.359644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.351885    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-podinfo\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.365442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.365408    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-token\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.378734 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.378670    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxjjb\" (UniqueName: \"kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-kube-api-access-sxjjb\") pod \"e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") " pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.413845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.407470    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:44.780587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.776045    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"]
Oct 30 14:39:44.811909 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.809227    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:39:44.811909 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.809324    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:39:44.811909 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:44.809365    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:39:45.183333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:45.183290    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" event={"ID":"d57850de-5a13-4f64-9e13-004cbd6eb898","Type":"ContainerStarted","Data":"12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6"}
Oct 30 14:39:45.183838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:45.183358    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" event={"ID":"d57850de-5a13-4f64-9e13-004cbd6eb898","Type":"ContainerStarted","Data":"511f9be78f3dbccb5ba1eedb5424b50885757d7f4317c2787c82f538ea4cd685"}
Oct 30 14:39:45.245578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:45.245511    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" podStartSLOduration=1.2454994369999999 podStartE2EDuration="1.245499437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:45.241194127 +0000 UTC m=+3050.273093066" watchObservedRunningTime="2026-10-30 14:39:45.245499437 +0000 UTC m=+3050.277398375"
Oct 30 14:39:45.416092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:45.416053    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:45.422929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:45.422905    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:46.190678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:46.190647    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:46.191593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:46.191574    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:39:54.898969 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:54.898919    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4" podUID="2a08e635-97e7-4723-94ae-a1c3b420fb52" containerName="main" containerID="cri-o://12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed" gracePeriod=45
Oct 30 14:39:55.186262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.186235    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:39:55.263828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.262042    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a08e635-97e7-4723-94ae-a1c3b420fb52-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a08e635-97e7-4723-94ae-a1c3b420fb52-tls-certs\") pod \"2a08e635-97e7-4723-94ae-a1c3b420fb52\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") "
Oct 30 14:39:55.263828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.262109    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-home\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-home\") pod \"2a08e635-97e7-4723-94ae-a1c3b420fb52\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") "
Oct 30 14:39:55.263828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.262196    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-model-cache\") pod \"2a08e635-97e7-4723-94ae-a1c3b420fb52\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") "
Oct 30 14:39:55.263828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.262300    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-dshm\") pod \"2a08e635-97e7-4723-94ae-a1c3b420fb52\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") "
Oct 30 14:39:55.263828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.262345    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-tmp-dir\") pod \"2a08e635-97e7-4723-94ae-a1c3b420fb52\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") "
Oct 30 14:39:55.263828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.262459    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a08e635-97e7-4723-94ae-a1c3b420fb52-kube-api-access-fr5nb\" (UniqueName: \"kubernetes.io/projected/2a08e635-97e7-4723-94ae-a1c3b420fb52-kube-api-access-fr5nb\") pod \"2a08e635-97e7-4723-94ae-a1c3b420fb52\" (UID: \"2a08e635-97e7-4723-94ae-a1c3b420fb52\") "
Oct 30 14:39:55.285396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.285076    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-tmp-dir" pod "2a08e635-97e7-4723-94ae-a1c3b420fb52" (UID: "2a08e635-97e7-4723-94ae-a1c3b420fb52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:39:55.285710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.285684    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a08e635-97e7-4723-94ae-a1c3b420fb52-tls-certs" pod "2a08e635-97e7-4723-94ae-a1c3b420fb52" (UID: "2a08e635-97e7-4723-94ae-a1c3b420fb52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:39:55.288298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.288267    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-home" pod "2a08e635-97e7-4723-94ae-a1c3b420fb52" (UID: "2a08e635-97e7-4723-94ae-a1c3b420fb52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:39:55.289381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.288693    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-model-cache" pod "2a08e635-97e7-4723-94ae-a1c3b420fb52" (UID: "2a08e635-97e7-4723-94ae-a1c3b420fb52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:39:55.298574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.295700    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a08e635-97e7-4723-94ae-a1c3b420fb52-kube-api-access-fr5nb" pod "2a08e635-97e7-4723-94ae-a1c3b420fb52" (UID: "2a08e635-97e7-4723-94ae-a1c3b420fb52"). InnerVolumeSpecName "kube-api-access-fr5nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:39:55.321234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.303281    4530 generic.go:356] "Generic (PLEG): container finished" podID="2a08e635-97e7-4723-94ae-a1c3b420fb52" containerID="12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed" exitCode=0
Oct 30 14:39:55.321234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.303490    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4" event={"ID":"2a08e635-97e7-4723-94ae-a1c3b420fb52","Type":"ContainerDied","Data":"12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed"}
Oct 30 14:39:55.321234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.303522    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4" event={"ID":"2a08e635-97e7-4723-94ae-a1c3b420fb52","Type":"ContainerDied","Data":"56a9d56c5566ddbc9271ef633be97be1bd9edffcacb7a5fe9b6f7e64f0445b27"}
Oct 30 14:39:55.321234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.303556    4530 scope.go:122] "RemoveContainer" containerID="12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed"
Oct 30 14:39:55.321234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.303719    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"
Oct 30 14:39:55.321234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.304777    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-dshm" pod "2a08e635-97e7-4723-94ae-a1c3b420fb52" (UID: "2a08e635-97e7-4723-94ae-a1c3b420fb52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:39:55.341660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.341636    4530 scope.go:122] "RemoveContainer" containerID="12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed"
Oct 30 14:39:55.342030 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:39:55.341997    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed\": container with ID starting with 12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed not found: ID does not exist" containerID="12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed"
Oct 30 14:39:55.342137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.342040    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed"} err="failed to get container status \"12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed\": rpc error: code = NotFound desc = could not find container \"12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed\": container with ID starting with 12cfb1f9f3ae918e9913c9afcb40ff72448c53d42e8be3b55b7f8105aa9da8ed not found: ID does not exist"
Oct 30 14:39:55.365167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.365139    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr5nb\" (UniqueName: \"kubernetes.io/projected/2a08e635-97e7-4723-94ae-a1c3b420fb52-kube-api-access-fr5nb\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:39:55.365167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.365165    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a08e635-97e7-4723-94ae-a1c3b420fb52-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:39:55.365316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.365174    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:39:55.365316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.365183    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:39:55.365316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.365190    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:39:55.365316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.365197    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a08e635-97e7-4723-94ae-a1c3b420fb52-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:39:55.659973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.659941    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"]
Oct 30 14:39:55.673884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:55.673854    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-396d70/sub-test-model-396d70-kserve-867db99ff9-t9dk4"]
Oct 30 14:39:57.149680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:57.149642    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a08e635-97e7-4723-94ae-a1c3b420fb52" path="/var/lib/kubelet/pods/2a08e635-97e7-4723-94ae-a1c3b420fb52/volumes"
Oct 30 14:39:57.849837 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:39:57.849798    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"]
Oct 30 14:40:05.454021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:05.453982    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:40:11.370141 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:40:11.370103    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:40:12.871318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:12.871261    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z" podUID="e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" containerName="main" containerID="cri-o://1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e" gracePeriod=45
Oct 30 14:40:13.014692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.014658    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z" podUID="e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" containerName="main" probeResult="failure" output="Get \"https://10.133.1.11:8000/ready\": dial tcp 10.133.1.11:8000: connect: connection refused"
Oct 30 14:40:13.220875 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.220844    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:40:13.321242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.321201    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-dshm\") pod \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") "
Oct 30 14:40:13.321457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.321435    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tmp-dir\") pod \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") "
Oct 30 14:40:13.321614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.321595    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-home\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-home\") pod \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") "
Oct 30 14:40:13.321695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.321646    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tls-certs\") pod \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") "
Oct 30 14:40:13.321757 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.321703    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-model-cache\") pod \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") "
Oct 30 14:40:13.321821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.321741    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tmp-dir" pod "e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" (UID: "e3bbdd89-e66c-4a3f-baa5-d1eace65f47c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:40:13.321911 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.321863    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-home" pod "e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" (UID: "e3bbdd89-e66c-4a3f-baa5-d1eace65f47c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:40:13.321911 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.321872    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-kube-api-access-p6d75\" (UniqueName: \"kubernetes.io/projected/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-kube-api-access-p6d75\") pod \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\" (UID: \"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c\") "
Oct 30 14:40:13.322058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.322034    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-model-cache" pod "e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" (UID: "e3bbdd89-e66c-4a3f-baa5-d1eace65f47c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:40:13.322698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.322669    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:40:13.322819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.322700    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:40:13.322819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.322714    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:40:13.323845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.323821    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-dshm" pod "e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" (UID: "e3bbdd89-e66c-4a3f-baa5-d1eace65f47c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:40:13.323981 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.323896    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tls-certs" pod "e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" (UID: "e3bbdd89-e66c-4a3f-baa5-d1eace65f47c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:40:13.324208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.324183    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-kube-api-access-p6d75" pod "e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" (UID: "e3bbdd89-e66c-4a3f-baa5-d1eace65f47c"). InnerVolumeSpecName "kube-api-access-p6d75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:40:13.423865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.423783    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6d75\" (UniqueName: \"kubernetes.io/projected/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-kube-api-access-p6d75\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:40:13.423865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.423812    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:40:13.423865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.423824    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:40:13.514900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.514865    4530 generic.go:356] "Generic (PLEG): container finished" podID="e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" containerID="1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e" exitCode=0
Oct 30 14:40:13.515084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.514916    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"
Oct 30 14:40:13.515084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.514975    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z" event={"ID":"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c","Type":"ContainerDied","Data":"1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e"}
Oct 30 14:40:13.515084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.515023    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z" event={"ID":"e3bbdd89-e66c-4a3f-baa5-d1eace65f47c","Type":"ContainerDied","Data":"aeb1191c606d0e1e8edef20bfdddb1490060315ec9ee10e93d292c8d0b04cc5c"}
Oct 30 14:40:13.515084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.515041    4530 scope.go:122] "RemoveContainer" containerID="1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e"
Oct 30 14:40:13.528285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.528266    4530 scope.go:122] "RemoveContainer" containerID="1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e"
Oct 30 14:40:13.528596 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:40:13.528575    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e\": container with ID starting with 1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e not found: ID does not exist" containerID="1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e"
Oct 30 14:40:13.528652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.528604    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e"} err="failed to get container status \"1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e\": rpc error: code = NotFound desc = could not find container \"1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e\": container with ID starting with 1976ee70cc5b7b6d0c479dec9b41f3dc3f8ac6a45249fbe2be1539479022426e not found: ID does not exist"
Oct 30 14:40:13.577425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.577394    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"]
Oct 30 14:40:13.606220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:13.606191    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6e91df/sub-test-model-6e91df-kserve-645cdcd7f9-kxc8z"]
Oct 30 14:40:15.117004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:15.116969    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" path="/var/lib/kubelet/pods/e3bbdd89-e66c-4a3f-baa5-d1eace65f47c/volumes"
Oct 30 14:40:18.556874 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:18.556838    4530 generic.go:356] "Generic (PLEG): container finished" podID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerID="dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612" exitCode=0
Oct 30 14:40:18.557438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:18.556913    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" event={"ID":"6a06488b-fc85-45f1-bbf9-a40a39125e5c","Type":"ContainerDied","Data":"dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612"}
Oct 30 14:40:18.557438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:18.557429    4530 scope.go:122] "RemoveContainer" containerID="dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612"
Oct 30 14:40:18.559244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:18.559212    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:40:18.559345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:18.559285    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:40:18.559345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:18.559322    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:40:19.567045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:19.567007    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" event={"ID":"6a06488b-fc85-45f1-bbf9-a40a39125e5c","Type":"ContainerStarted","Data":"665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a"}
Oct 30 14:40:19.799173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:19.799136    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:40:19.804610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:19.803637    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 30 14:40:19.804610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:19.803712    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 30 14:40:20.794167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:20.794126    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 30 14:40:20.794809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:20.794776    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 30 14:40:21.153844 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:21.153812    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:40:21.799954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:21.799915    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 30 14:40:21.800583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:21.800532    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 30 14:40:22.794162 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:22.794123    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 30 14:40:22.795625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:22.795589    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 30 14:40:23.792574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:23.792517    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 30 14:40:23.794176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:23.794142    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 30 14:40:24.784385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:24.784346    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 30 14:40:24.784612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:24.784409    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 30 14:40:25.789711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:25.789676    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:40:25.790956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:25.790930    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:40:26.740567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.740518    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"]
Oct 30 14:40:26.741244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.741224    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" containerName="main"
Oct 30 14:40:26.741244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.741245    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" containerName="main"
Oct 30 14:40:26.741393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.741277    4530 cpu_manager.go:414] "containerMap: removing container" podUID="2a08e635-97e7-4723-94ae-a1c3b420fb52" containerName="main"
Oct 30 14:40:26.741393 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.741284    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a08e635-97e7-4723-94ae-a1c3b420fb52" containerName="main"
Oct 30 14:40:26.741467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.741399    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3bbdd89-e66c-4a3f-baa5-d1eace65f47c" containerName="main"
Oct 30 14:40:26.741467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.741412    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a08e635-97e7-4723-94ae-a1c3b420fb52" containerName="main"
Oct 30 14:40:26.746003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.745976    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.767552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.767513    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"]
Oct 30 14:40:26.777083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.777007    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e50e63\"/\"auth-test-model-e50e63-kserve-self-signed-certs\""
Oct 30 14:40:26.777083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.777015    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e50e63\"/\"default-dockercfg-5w5jm\""
Oct 30 14:40:26.781878 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.781853    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e50e63\"/\"openshift-service-ca.crt\""
Oct 30 14:40:26.782003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.781971    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e50e63\"/\"kube-root-ca.crt\""
Oct 30 14:40:26.865899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.865860    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-model-cache\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.866338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.865903    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-dshm\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.866338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.866040    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jph52\" (UniqueName: \"kubernetes.io/projected/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-kube-api-access-jph52\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.866338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.866080    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-home\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.866338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.866146    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tmp-dir\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.866519 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.866337    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tls-certs\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.987166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.985766    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tls-certs\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.987166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.985960    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-model-cache\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.987166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.985993    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-dshm\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.987166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.986071    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jph52\" (UniqueName: \"kubernetes.io/projected/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-kube-api-access-jph52\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.987166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.986108    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-home\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.987166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.986144    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tmp-dir\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:26.987166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.986726    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tmp-dir\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:27.000710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:26.992201    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-home\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:27.000710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:27.000576    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-model-cache\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:27.001633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:27.001581    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tls-certs\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:27.009832 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:27.009807    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-dshm\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:27.042068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:27.042025    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jph52\" (UniqueName: \"kubernetes.io/projected/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-kube-api-access-jph52\") pod \"auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") " pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:27.068017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:27.067978    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:27.428593 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:40:27.428529    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8ba4e6e_78fb_418f_a4c0_7b0f3d9871cc.slice/crio-5865df31c647a1addb2f028d4fba6b5a7672858be7c20577680ae8aca969d21a WatchSource:0}: Error finding container 5865df31c647a1addb2f028d4fba6b5a7672858be7c20577680ae8aca969d21a: Status 404 returned error can't find the container with id 5865df31c647a1addb2f028d4fba6b5a7672858be7c20577680ae8aca969d21a
Oct 30 14:40:27.430881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:27.430856    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"]
Oct 30 14:40:27.677399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:27.677311    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz" event={"ID":"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc","Type":"ContainerStarted","Data":"f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01"}
Oct 30 14:40:27.677399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:27.677360    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz" event={"ID":"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc","Type":"ContainerStarted","Data":"5865df31c647a1addb2f028d4fba6b5a7672858be7c20577680ae8aca969d21a"}
Oct 30 14:40:27.729482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:27.729427    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz" podStartSLOduration=1.729411594 podStartE2EDuration="1.729411594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:27.727384551 +0000 UTC m=+3092.759283490" watchObservedRunningTime="2026-10-30 14:40:27.729411594 +0000 UTC m=+3092.761310556"
Oct 30 14:40:28.069323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:28.069293    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:37.069714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:37.069669    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:37.081945 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:37.081917    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:37.765574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:37.765481    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:40:41.368624 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:40:41.368590    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:40:47.501732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.501684    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"]
Oct 30 14:40:47.507788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.507758    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.528163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.528137    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"]
Oct 30 14:40:47.536882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.536847    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2927fd-openshift-default-dockercfg-v74nw\""
Oct 30 14:40:47.572484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.572446    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-certs\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.572668 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.572515    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-credential-socket\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.572812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.572773    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istiod-ca-cert\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.572878 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.572829    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-socket\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.573010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.572987    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-token\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.573078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.573026    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-data\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.573141 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.573111    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-envoy\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.573277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.573255    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-podinfo\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.573317 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.573304    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k97wx\" (UniqueName: \"kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-kube-api-access-k97wx\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.677482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.677430    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-envoy\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.678489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.678458    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-podinfo\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.678823 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.678799    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k97wx\" (UniqueName: \"kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-kube-api-access-k97wx\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.679002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.678982    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-certs\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.679166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.679146    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-credential-socket\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.679490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.679468    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istiod-ca-cert\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.683687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.683664    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-socket\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.683903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.683883    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-token\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.684028 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.684012    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-data\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.705341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.684454    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-credential-socket\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.705561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.685041    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-certs\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.705694 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.685286    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-socket\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.705797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.685927    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istiod-ca-cert\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.705893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.690839    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-data\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.705990 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.696943    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-envoy\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.706097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.704121    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-podinfo\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.715717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.715691    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-token\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.740909 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.740825    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k97wx\" (UniqueName: \"kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-kube-api-access-k97wx\") pod \"e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") " pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:47.838817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:47.838779    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:48.239439 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:40:48.239402    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c09cc6a_b1c6_4dba_a559_1f957e3d3e4d.slice/crio-a7335c1c53186b0bce84773ceb8f4936aec9d6433f657b3fb75c8729ee9b9a40 WatchSource:0}: Error finding container a7335c1c53186b0bce84773ceb8f4936aec9d6433f657b3fb75c8729ee9b9a40: Status 404 returned error can't find the container with id a7335c1c53186b0bce84773ceb8f4936aec9d6433f657b3fb75c8729ee9b9a40
Oct 30 14:40:48.240036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:48.240007    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"]
Oct 30 14:40:48.242209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:48.242174    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:40:48.242293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:48.242244    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:40:48.242293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:48.242276    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:40:48.903705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:48.903613    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" event={"ID":"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d","Type":"ContainerStarted","Data":"7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649"}
Oct 30 14:40:48.903705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:48.903685    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" event={"ID":"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d","Type":"ContainerStarted","Data":"a7335c1c53186b0bce84773ceb8f4936aec9d6433f657b3fb75c8729ee9b9a40"}
Oct 30 14:40:48.974935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:48.974883    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" podStartSLOduration=1.974869955 podStartE2EDuration="1.974869955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:48.959386333 +0000 UTC m=+3113.991285272" watchObservedRunningTime="2026-10-30 14:40:48.974869955 +0000 UTC m=+3114.006768893"
Oct 30 14:40:49.839957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:49.839916    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:49.846115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:49.846086    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:49.912239 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:49.912207    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:49.912931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:49.912911    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:40:57.581359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.581325    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"]
Oct 30 14:40:57.585731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.585712    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"
Oct 30 14:40:57.623183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.623149    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0e8b9-dockercfg-zzkbk\""
Oct 30 14:40:57.623357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.623166    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0e8b9\""
Oct 30 14:40:57.629859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.629831    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"]
Oct 30 14:40:57.685246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.685220    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0852550-3a42-4c43-9c97-c73ac997c4d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn\" (UID: \"c0852550-3a42-4c43-9c97-c73ac997c4d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"
Oct 30 14:40:57.685438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.685412    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qmkd\" (UniqueName: \"kubernetes.io/projected/c0852550-3a42-4c43-9c97-c73ac997c4d7-kube-api-access-6qmkd\") pod \"payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn\" (UID: \"c0852550-3a42-4c43-9c97-c73ac997c4d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"
Oct 30 14:40:57.735312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.735256    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"]
Oct 30 14:40:57.741220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.741196    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"
Oct 30 14:40:57.742427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.742399    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"]
Oct 30 14:40:57.786292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.786263    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0852550-3a42-4c43-9c97-c73ac997c4d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn\" (UID: \"c0852550-3a42-4c43-9c97-c73ac997c4d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"
Oct 30 14:40:57.786435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.786416    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f\" (UID: \"d4ce80eb-7159-47c6-a5b5-cf94871d1e46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"
Oct 30 14:40:57.786619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.786602    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qmkd\" (UniqueName: \"kubernetes.io/projected/c0852550-3a42-4c43-9c97-c73ac997c4d7-kube-api-access-6qmkd\") pod \"payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn\" (UID: \"c0852550-3a42-4c43-9c97-c73ac997c4d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"
Oct 30 14:40:57.786732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.786711    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxls\" (UniqueName: \"kubernetes.io/projected/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-kube-api-access-fwxls\") pod \"payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f\" (UID: \"d4ce80eb-7159-47c6-a5b5-cf94871d1e46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"
Oct 30 14:40:57.787205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.787181    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0852550-3a42-4c43-9c97-c73ac997c4d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn\" (UID: \"c0852550-3a42-4c43-9c97-c73ac997c4d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"
Oct 30 14:40:57.813962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.813929    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qmkd\" (UniqueName: \"kubernetes.io/projected/c0852550-3a42-4c43-9c97-c73ac997c4d7-kube-api-access-6qmkd\") pod \"payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn\" (UID: \"c0852550-3a42-4c43-9c97-c73ac997c4d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"
Oct 30 14:40:57.896510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.896417    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f\" (UID: \"d4ce80eb-7159-47c6-a5b5-cf94871d1e46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"
Oct 30 14:40:57.914795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.913524    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxls\" (UniqueName: \"kubernetes.io/projected/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-kube-api-access-fwxls\") pod \"payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f\" (UID: \"d4ce80eb-7159-47c6-a5b5-cf94871d1e46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"
Oct 30 14:40:57.914795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.913170    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"
Oct 30 14:40:57.915097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.898592    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f\" (UID: \"d4ce80eb-7159-47c6-a5b5-cf94871d1e46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"
Oct 30 14:40:57.981555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:57.981501    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxls\" (UniqueName: \"kubernetes.io/projected/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-kube-api-access-fwxls\") pod \"payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f\" (UID: \"d4ce80eb-7159-47c6-a5b5-cf94871d1e46\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"
Oct 30 14:40:58.057476 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:58.057432    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"
Oct 30 14:40:58.456428 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:40:58.456381    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0852550_3a42_4c43_9c97_c73ac997c4d7.slice/crio-8a88ff948991fdc43e3069e9331ce4d1a38f081c8b516f3ef53611549ab7e4de WatchSource:0}: Error finding container 8a88ff948991fdc43e3069e9331ce4d1a38f081c8b516f3ef53611549ab7e4de: Status 404 returned error can't find the container with id 8a88ff948991fdc43e3069e9331ce4d1a38f081c8b516f3ef53611549ab7e4de
Oct 30 14:40:58.457274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:58.457183    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"]
Oct 30 14:40:58.478359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:58.470654    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"]
Oct 30 14:40:58.490977 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:40:58.490923    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ce80eb_7159_47c6_a5b5_cf94871d1e46.slice/crio-9dbd2405f4401336c920decdb58edeb4c50e9e2e5bc0ba0f69d32531de74b633 WatchSource:0}: Error finding container 9dbd2405f4401336c920decdb58edeb4c50e9e2e5bc0ba0f69d32531de74b633: Status 404 returned error can't find the container with id 9dbd2405f4401336c920decdb58edeb4c50e9e2e5bc0ba0f69d32531de74b633
Oct 30 14:40:59.070138 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:59.070100    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f" event={"ID":"d4ce80eb-7159-47c6-a5b5-cf94871d1e46","Type":"ContainerStarted","Data":"77306de49e7b8c6a7cc05f6479bcb52a91ca7af813026a29449dca9d6e52f227"}
Oct 30 14:40:59.070692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:59.070171    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f" event={"ID":"d4ce80eb-7159-47c6-a5b5-cf94871d1e46","Type":"ContainerStarted","Data":"9dbd2405f4401336c920decdb58edeb4c50e9e2e5bc0ba0f69d32531de74b633"}
Oct 30 14:40:59.070692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:59.070196    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"
Oct 30 14:40:59.072265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:59.072238    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn" event={"ID":"c0852550-3a42-4c43-9c97-c73ac997c4d7","Type":"ContainerStarted","Data":"42a77c50d27fd74786502cbfb94e0f086897f99cceb6667d65895de8684730b2"}
Oct 30 14:40:59.072406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:59.072283    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn" event={"ID":"c0852550-3a42-4c43-9c97-c73ac997c4d7","Type":"ContainerStarted","Data":"8a88ff948991fdc43e3069e9331ce4d1a38f081c8b516f3ef53611549ab7e4de"}
Oct 30 14:40:59.072406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:59.072316    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"
Oct 30 14:40:59.126242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:59.126185    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f" podStartSLOduration=2.126168309 podStartE2EDuration="2.126168309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:59.12005172 +0000 UTC m=+3124.151950662" watchObservedRunningTime="2026-10-30 14:40:59.126168309 +0000 UTC m=+3124.158067254"
Oct 30 14:40:59.169469 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:40:59.169420    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn" podStartSLOduration=2.169406996 podStartE2EDuration="2.169406996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:59.167750848 +0000 UTC m=+3124.199649810" watchObservedRunningTime="2026-10-30 14:40:59.169406996 +0000 UTC m=+3124.201306015"
Oct 30 14:41:00.417876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.417837    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"]
Oct 30 14:41:00.422622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.422604    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.447033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.446340    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"]
Oct 30 14:41:00.449306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.449279    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a1854b-serving-cert\""
Oct 30 14:41:00.583615 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.583578    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr74f\" (UniqueName: \"kubernetes.io/projected/72181d49-18db-4413-9c82-82d995de0b43-kube-api-access-tr74f\") pod \"maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.583796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.583771    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.583851 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.583815    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72181d49-18db-4413-9c82-82d995de0b43-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.583985 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.583961    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-metrics-certs\") pod \"maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.687905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.686885    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72181d49-18db-4413-9c82-82d995de0b43-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.687905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.687058    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-metrics-certs\") pod \"maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.687905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.687207    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr74f\" (UniqueName: \"kubernetes.io/projected/72181d49-18db-4413-9c82-82d995de0b43-kube-api-access-tr74f\") pod \"maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.687905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.687401    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.698576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.693007    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72181d49-18db-4413-9c82-82d995de0b43-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.703634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.699433    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-metrics-certs\") pod \"maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.718667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.716343    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.733563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.723621    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr74f\" (UniqueName: \"kubernetes.io/projected/72181d49-18db-4413-9c82-82d995de0b43-kube-api-access-tr74f\") pod \"maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:00.748567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:00.741826    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:01.078977 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:41:01.078946    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72181d49_18db_4413_9c82_82d995de0b43.slice/crio-bb9264a33fa4f5fc050c002a86ee0c9a47ec813cacec5d151f1145a807c519b1 WatchSource:0}: Error finding container bb9264a33fa4f5fc050c002a86ee0c9a47ec813cacec5d151f1145a807c519b1: Status 404 returned error can't find the container with id bb9264a33fa4f5fc050c002a86ee0c9a47ec813cacec5d151f1145a807c519b1
Oct 30 14:41:01.080878 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:01.080858    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:41:01.081357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:01.081333    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"]
Oct 30 14:41:01.091832 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:01.091800    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk" event={"ID":"72181d49-18db-4413-9c82-82d995de0b43","Type":"ContainerStarted","Data":"bb9264a33fa4f5fc050c002a86ee0c9a47ec813cacec5d151f1145a807c519b1"}
Oct 30 14:41:02.129288 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:02.129239    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk" event={"ID":"72181d49-18db-4413-9c82-82d995de0b43","Type":"ContainerStarted","Data":"1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90"}
Oct 30 14:41:02.129828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:02.129513    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:02.184034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:02.183982    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk" podStartSLOduration=1.833259404 podStartE2EDuration="2.183969088s" totalImagesPullingTime="350.709684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:41:02.178654737 +0000 UTC m=+3127.210553676" watchObservedRunningTime="2026-10-30 14:41:02.183969088 +0000 UTC m=+3127.215868027"
Oct 30 14:41:08.142529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:08.142498    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:41:10.083610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:10.083579    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"
Oct 30 14:41:10.084105 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:10.083668    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"
Oct 30 14:41:11.368374 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:41:11.368331    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:41:13.058568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.051829    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"]
Oct 30 14:41:13.141587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.141554    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"]
Oct 30 14:41:13.172033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.144073    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.203614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.203583    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3b95dd-openshift-default-dockercfg-vrcsf\""
Oct 30 14:41:13.245217 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.245173    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-credential-socket\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.257928 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.257895    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-certs\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.262489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.262452    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-token\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.262944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.262915    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-podinfo\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.263216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.263197    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-envoy\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.265233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.265209    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ddd71b-1fbf-4359-b8a3-da951b41869b-istiod-ca-cert\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.265482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.265463    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjhdb\" (UniqueName: \"kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-kube-api-access-mjhdb\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.265821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.265803    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-data\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.266164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.266139    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-socket\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368196 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368110    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjhdb\" (UniqueName: \"kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-kube-api-access-mjhdb\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368229    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-data\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368274    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-socket\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368318    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-credential-socket\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368346    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-certs\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368459    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-token\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368625 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368578    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-podinfo\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368655    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-envoy\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368683    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ddd71b-1fbf-4359-b8a3-da951b41869b-istiod-ca-cert\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368737    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-data\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368776    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-socket\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368871    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-credential-socket\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.368936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.368906    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-certs\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.369397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.369373    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ddd71b-1fbf-4359-b8a3-da951b41869b-istiod-ca-cert\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.370939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.370921    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-envoy\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.371272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.371248    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-podinfo\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.417057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.417021    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-token\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.417203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.417095    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjhdb\" (UniqueName: \"kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-kube-api-access-mjhdb\") pod \"e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") " pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.534653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.534600    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:13.860093 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:41:13.860050    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ddd71b_1fbf_4359_b8a3_da951b41869b.slice/crio-db37ae6dfcfb7ca21fe02ec584baa984b4d99517d1d7f5b2892b3f6322891635 WatchSource:0}: Error finding container db37ae6dfcfb7ca21fe02ec584baa984b4d99517d1d7f5b2892b3f6322891635: Status 404 returned error can't find the container with id db37ae6dfcfb7ca21fe02ec584baa984b4d99517d1d7f5b2892b3f6322891635
Oct 30 14:41:13.861819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.861795    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"]
Oct 30 14:41:13.863265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.863231    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:41:13.863336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.863300    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:41:13.863377 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:13.863335    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:41:14.278351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:14.278255    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" event={"ID":"55ddd71b-1fbf-4359-b8a3-da951b41869b","Type":"ContainerStarted","Data":"f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa"}
Oct 30 14:41:14.278351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:14.278322    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" event={"ID":"55ddd71b-1fbf-4359-b8a3-da951b41869b","Type":"ContainerStarted","Data":"db37ae6dfcfb7ca21fe02ec584baa984b4d99517d1d7f5b2892b3f6322891635"}
Oct 30 14:41:14.343501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:14.343435    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" podStartSLOduration=2.343414444 podStartE2EDuration="2.343414444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:41:14.330011081 +0000 UTC m=+3139.361910057" watchObservedRunningTime="2026-10-30 14:41:14.343414444 +0000 UTC m=+3139.375313386"
Oct 30 14:41:14.535779 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:14.535699    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:14.541984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:14.541953    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:15.286020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:15.285978    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:15.286998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:15.286975    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:41:26.365565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:26.365500    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"]
Oct 30 14:41:26.371327 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:26.371306    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"
Oct 30 14:41:26.395638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:26.395612    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"]
Oct 30 14:41:26.403352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:26.403323    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-460b0b-dockercfg-8z7s6\""
Oct 30 14:41:26.403579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:26.403557    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-460b0b\""
Oct 30 14:41:26.443762 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:26.443733    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n\" (UID: \"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"
Oct 30 14:41:26.443979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:26.443956    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwn57\" (UniqueName: \"kubernetes.io/projected/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-kube-api-access-pwn57\") pod \"payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n\" (UID: \"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"
Oct 30 14:41:26.544892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:26.544853    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n\" (UID: \"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"
Oct 30 14:41:26.545031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:26.545015    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwn57\" (UniqueName: \"kubernetes.io/projected/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-kube-api-access-pwn57\") pod \"payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n\" (UID: \"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"
Oct 30 14:41:26.545602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:26.545577    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n\" (UID: \"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"
Oct 30 14:41:26.573605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:26.573580    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwn57\" (UniqueName: \"kubernetes.io/projected/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-kube-api-access-pwn57\") pod \"payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n\" (UID: \"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"
Oct 30 14:41:26.687028 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:26.686962    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"
Oct 30 14:41:27.004139 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:41:27.004109    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61dcc1f8_e09e_463f_8c94_f8453bfdcf1f.slice/crio-125ff0ed4e291deb7abd8321c9f7d46a991ad8e3f7118e8ac1e55a4fa9b4e279 WatchSource:0}: Error finding container 125ff0ed4e291deb7abd8321c9f7d46a991ad8e3f7118e8ac1e55a4fa9b4e279: Status 404 returned error can't find the container with id 125ff0ed4e291deb7abd8321c9f7d46a991ad8e3f7118e8ac1e55a4fa9b4e279
Oct 30 14:41:27.006632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:27.006608    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"]
Oct 30 14:41:27.387302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:27.387265    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n" event={"ID":"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f","Type":"ContainerStarted","Data":"7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83"}
Oct 30 14:41:27.387725 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:27.387324    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n" event={"ID":"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f","Type":"ContainerStarted","Data":"125ff0ed4e291deb7abd8321c9f7d46a991ad8e3f7118e8ac1e55a4fa9b4e279"}
Oct 30 14:41:27.387725 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:27.387399    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"
Oct 30 14:41:27.449827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:27.449776    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n" podStartSLOduration=1.449763076 podStartE2EDuration="1.449763076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:41:27.445928531 +0000 UTC m=+3152.477827477" watchObservedRunningTime="2026-10-30 14:41:27.449763076 +0000 UTC m=+3152.481662015"
Oct 30 14:41:34.052572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:34.049979    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:41:38.405751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:38.405700    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"
Oct 30 14:41:41.368387 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:41:41.368348    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:41:46.192632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:46.192597    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 30 14:41:46.193069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:46.192653    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 30 14:41:46.699534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:46.699494    4530 generic.go:356] "Generic (PLEG): container finished" podID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerID="12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6" exitCode=0
Oct 30 14:41:46.699754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:46.699587    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" event={"ID":"d57850de-5a13-4f64-9e13-004cbd6eb898","Type":"ContainerDied","Data":"12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6"}
Oct 30 14:41:46.700068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:46.700049    4530 scope.go:122] "RemoveContainer" containerID="12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6"
Oct 30 14:41:46.702035 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:46.702007    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:41:46.702120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:46.702078    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:41:46.702120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:46.702111    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:41:47.714195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:47.714157    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" event={"ID":"d57850de-5a13-4f64-9e13-004cbd6eb898","Type":"ContainerStarted","Data":"a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a"}
Oct 30 14:41:48.415917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:48.415884    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:41:48.416102 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:48.416053    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 30 14:41:48.416162 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:48.416108    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 30 14:41:49.416666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:49.416613    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 30 14:41:49.417088 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:49.416686    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 30 14:41:50.416319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:50.416259    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 30 14:41:50.416511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:50.416335    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 30 14:41:51.416208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:51.416173    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 30 14:41:51.416769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:51.416231    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 30 14:41:52.420042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:52.420003    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body=
Oct 30 14:41:52.420686 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:52.420652    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused"
Oct 30 14:41:53.421800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:53.421765    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:41:53.422280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:53.422257    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:41:53.423062 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:53.423045    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:41:57.486775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:57.486746    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"]
Oct 30 14:41:57.494043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:57.494009    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"]
Oct 30 14:41:57.494169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:57.494109    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"
Oct 30 14:41:57.526220 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:57.526162    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-33c209\""
Oct 30 14:41:57.526376 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:57.526300    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-33c209-dockercfg-26qrr\""
Oct 30 14:41:57.663227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:57.663179    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7\" (UID: \"18d70ff9-ac45-4052-a9a2-e9a6bd0af192\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"
Oct 30 14:41:57.663424 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:57.663391    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twlsq\" (UniqueName: \"kubernetes.io/projected/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-kube-api-access-twlsq\") pod \"payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7\" (UID: \"18d70ff9-ac45-4052-a9a2-e9a6bd0af192\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"
Oct 30 14:41:57.764935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:57.764851    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7\" (UID: \"18d70ff9-ac45-4052-a9a2-e9a6bd0af192\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"
Oct 30 14:41:57.765096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:57.764967    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twlsq\" (UniqueName: \"kubernetes.io/projected/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-kube-api-access-twlsq\") pod \"payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7\" (UID: \"18d70ff9-ac45-4052-a9a2-e9a6bd0af192\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"
Oct 30 14:41:57.765656 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:57.765633    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7\" (UID: \"18d70ff9-ac45-4052-a9a2-e9a6bd0af192\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"
Oct 30 14:41:57.794406 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:57.794379    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twlsq\" (UniqueName: \"kubernetes.io/projected/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-kube-api-access-twlsq\") pod \"payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7\" (UID: \"18d70ff9-ac45-4052-a9a2-e9a6bd0af192\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"
Oct 30 14:41:57.814601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:57.814582    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"
Oct 30 14:41:58.122981 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:41:58.122934    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d70ff9_ac45_4052_a9a2_e9a6bd0af192.slice/crio-9e6739736098569b9e9e38439a4bf1c887fb42dffc92ac54d97b77cfbc32bfb3 WatchSource:0}: Error finding container 9e6739736098569b9e9e38439a4bf1c887fb42dffc92ac54d97b77cfbc32bfb3: Status 404 returned error can't find the container with id 9e6739736098569b9e9e38439a4bf1c887fb42dffc92ac54d97b77cfbc32bfb3
Oct 30 14:41:58.124709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:58.124680    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"]
Oct 30 14:41:58.806532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:58.806493    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7" event={"ID":"18d70ff9-ac45-4052-a9a2-e9a6bd0af192","Type":"ContainerStarted","Data":"6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0"}
Oct 30 14:41:58.806956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:58.806601    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"
Oct 30 14:41:58.806956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:58.806625    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7" event={"ID":"18d70ff9-ac45-4052-a9a2-e9a6bd0af192","Type":"ContainerStarted","Data":"9e6739736098569b9e9e38439a4bf1c887fb42dffc92ac54d97b77cfbc32bfb3"}
Oct 30 14:41:58.871431 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:41:58.871377    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7" podStartSLOduration=1.8713646449999999 podStartE2EDuration="1.871364645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:41:58.868003206 +0000 UTC m=+3183.899902146" watchObservedRunningTime="2026-10-30 14:41:58.871364645 +0000 UTC m=+3183.903263584"
Oct 30 14:42:09.818567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:09.818459    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"
Oct 30 14:42:11.369524 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:42:11.369482    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:42:39.758068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:39.758022    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"]
Oct 30 14:42:40.915680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:40.915626    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:42:41.368579 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:42:41.368509    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:42:44.952443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:44.952402    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"]
Oct 30 14:42:50.953559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:50.952717    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:42:50.953559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:50.952820    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:42:52.258620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.258581    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"]
Oct 30 14:42:52.259113 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.258801    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62" podUID="1e5cbbcf-0058-44e5-8e80-2c4fb100bf70" containerName="maas-api" containerID="cri-o://261b440cf2b1db5ac1c16ad38002a6fbb3197c6fa1e2fce0c45b4bedd5e7b5ad" gracePeriod=30
Oct 30 14:42:52.437206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.437165    4530 generic.go:356] "Generic (PLEG): container finished" podID="1e5cbbcf-0058-44e5-8e80-2c4fb100bf70" containerID="261b440cf2b1db5ac1c16ad38002a6fbb3197c6fa1e2fce0c45b4bedd5e7b5ad" exitCode=0
Oct 30 14:42:52.438131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.437795    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62" event={"ID":"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70","Type":"ContainerDied","Data":"261b440cf2b1db5ac1c16ad38002a6fbb3197c6fa1e2fce0c45b4bedd5e7b5ad"}
Oct 30 14:42:52.447798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.447292    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"]
Oct 30 14:42:52.447971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.447890    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9" podUID="84c3cc86-1167-4863-8f71-2c66a873b96a" containerName="payload-processing" containerID="cri-o://3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c" gracePeriod=30
Oct 30 14:42:52.607173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.606870    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:42:52.700752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.700714    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-metrics-certs\") pod \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") "
Oct 30 14:42:52.701032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.700976    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-kube-api-access-xcdtp\" (UniqueName: \"kubernetes.io/projected/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-kube-api-access-xcdtp\") pod \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") "
Oct 30 14:42:52.701276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.701183    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-openshift-service-ca\") pod \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") "
Oct 30 14:42:52.701276 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.701218    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-maas-api-tls\") pod \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\" (UID: \"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70\") "
Oct 30 14:42:52.702675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.702643    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-openshift-service-ca" pod "1e5cbbcf-0058-44e5-8e80-2c4fb100bf70" (UID: "1e5cbbcf-0058-44e5-8e80-2c4fb100bf70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:42:52.705354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.705321    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-metrics-certs" pod "1e5cbbcf-0058-44e5-8e80-2c4fb100bf70" (UID: "1e5cbbcf-0058-44e5-8e80-2c4fb100bf70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:42:52.707893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.707867    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-kube-api-access-xcdtp" pod "1e5cbbcf-0058-44e5-8e80-2c4fb100bf70" (UID: "1e5cbbcf-0058-44e5-8e80-2c4fb100bf70"). InnerVolumeSpecName "kube-api-access-xcdtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:42:52.709701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.709658    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-maas-api-tls" pod "1e5cbbcf-0058-44e5-8e80-2c4fb100bf70" (UID: "1e5cbbcf-0058-44e5-8e80-2c4fb100bf70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:42:52.741396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.741360    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"
Oct 30 14:42:52.803043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.802951    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c3cc86-1167-4863-8f71-2c66a873b96a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c3cc86-1167-4863-8f71-2c66a873b96a-plugins-config-volume\") pod \"84c3cc86-1167-4863-8f71-2c66a873b96a\" (UID: \"84c3cc86-1167-4863-8f71-2c66a873b96a\") "
Oct 30 14:42:52.803226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.803087    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c3cc86-1167-4863-8f71-2c66a873b96a-kube-api-access-bt98h\" (UniqueName: \"kubernetes.io/projected/84c3cc86-1167-4863-8f71-2c66a873b96a-kube-api-access-bt98h\") pod \"84c3cc86-1167-4863-8f71-2c66a873b96a\" (UID: \"84c3cc86-1167-4863-8f71-2c66a873b96a\") "
Oct 30 14:42:52.803862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.803490    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcdtp\" (UniqueName: \"kubernetes.io/projected/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-kube-api-access-xcdtp\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:42:52.803862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.803518    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:42:52.803862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.803535    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:42:52.803862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.803561    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:42:52.803862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.803820    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c3cc86-1167-4863-8f71-2c66a873b96a-plugins-config-volume" pod "84c3cc86-1167-4863-8f71-2c66a873b96a" (UID: "84c3cc86-1167-4863-8f71-2c66a873b96a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:42:52.806031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.806007    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c3cc86-1167-4863-8f71-2c66a873b96a-kube-api-access-bt98h" pod "84c3cc86-1167-4863-8f71-2c66a873b96a" (UID: "84c3cc86-1167-4863-8f71-2c66a873b96a"). InnerVolumeSpecName "kube-api-access-bt98h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:42:52.933568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.924848    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c3cc86-1167-4863-8f71-2c66a873b96a-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:42:52.933568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:52.924897    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt98h\" (UniqueName: \"kubernetes.io/projected/84c3cc86-1167-4863-8f71-2c66a873b96a-kube-api-access-bt98h\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:42:53.448233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.448132    4530 generic.go:356] "Generic (PLEG): container finished" podID="84c3cc86-1167-4863-8f71-2c66a873b96a" containerID="3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c" exitCode=0
Oct 30 14:42:53.448676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.448264    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9" event={"ID":"84c3cc86-1167-4863-8f71-2c66a873b96a","Type":"ContainerDied","Data":"3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c"}
Oct 30 14:42:53.448676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.448308    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9" event={"ID":"84c3cc86-1167-4863-8f71-2c66a873b96a","Type":"ContainerDied","Data":"51ea6e95438135a54b2fe1bee82e8864c2e776177c1269438eb0b6082dae061c"}
Oct 30 14:42:53.448676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.448331    4530 scope.go:122] "RemoveContainer" containerID="3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c"
Oct 30 14:42:53.448676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.448469    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"
Oct 30 14:42:53.451788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.451760    4530 generic.go:356] "Generic (PLEG): container finished" podID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerID="f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa" exitCode=0
Oct 30 14:42:53.451923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.451797    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" event={"ID":"55ddd71b-1fbf-4359-b8a3-da951b41869b","Type":"ContainerDied","Data":"f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa"}
Oct 30 14:42:53.452316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.452296    4530 scope.go:122] "RemoveContainer" containerID="f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa"
Oct 30 14:42:53.453986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.453930    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62" event={"ID":"1e5cbbcf-0058-44e5-8e80-2c4fb100bf70","Type":"ContainerDied","Data":"63c9d321761c3fa44d356dc3e5f56b100e1ebd7cbffdab40574f4c735db7a28f"}
Oct 30 14:42:53.453986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.453950    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"
Oct 30 14:42:53.454852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.454824    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:42:53.454960 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.454918    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:42:53.455024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.454975    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:42:53.462153 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.462133    4530 scope.go:122] "RemoveContainer" containerID="3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c"
Oct 30 14:42:53.462479 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:42:53.462454    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c\": container with ID starting with 3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c not found: ID does not exist" containerID="3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c"
Oct 30 14:42:53.462607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.462492    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c"} err="failed to get container status \"3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c\": rpc error: code = NotFound desc = could not find container \"3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c\": container with ID starting with 3a25f07419965c9f96c1c03be7c11e3d3e7f3372c438973c769d12bd1340f53c not found: ID does not exist"
Oct 30 14:42:53.462607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.462517    4530 scope.go:122] "RemoveContainer" containerID="261b440cf2b1db5ac1c16ad38002a6fbb3197c6fa1e2fce0c45b4bedd5e7b5ad"
Oct 30 14:42:53.518818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.518792    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"]
Oct 30 14:42:53.538523 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.538495    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d3a829-5b7d7cf5d8-hgs62"]
Oct 30 14:42:53.590353 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.590321    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"]
Oct 30 14:42:53.598004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:53.597968    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3a829-77fcbf8b76-8q8h9"]
Oct 30 14:42:54.464950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:54.464912    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" event={"ID":"55ddd71b-1fbf-4359-b8a3-da951b41869b","Type":"ContainerStarted","Data":"f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac"}
Oct 30 14:42:54.467839 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:54.467808    4530 generic.go:356] "Generic (PLEG): container finished" podID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerID="7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649" exitCode=0
Oct 30 14:42:54.468005 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:54.467854    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" event={"ID":"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d","Type":"ContainerDied","Data":"7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649"}
Oct 30 14:42:54.468355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:54.468334    4530 scope.go:122] "RemoveContainer" containerID="7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649"
Oct 30 14:42:54.471189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:54.471153    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:42:54.471334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:54.471229    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:42:54.471334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:54.471264    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:42:54.535176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:54.535140    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:42:54.535370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:54.535349    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 14:42:54.535422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:54.535393    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 14:42:54.792623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:54.780951    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2" podUID="bcb56386-09e1-4fa0-9a80-f291fc293fd1" containerName="main" containerID="cri-o://98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41" gracePeriod=45
Oct 30 14:42:55.129804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.129272    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5cbbcf-0058-44e5-8e80-2c4fb100bf70" path="/var/lib/kubelet/pods/1e5cbbcf-0058-44e5-8e80-2c4fb100bf70/volumes"
Oct 30 14:42:55.130410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.130384    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c3cc86-1167-4863-8f71-2c66a873b96a" path="/var/lib/kubelet/pods/84c3cc86-1167-4863-8f71-2c66a873b96a/volumes"
Oct 30 14:42:55.132793 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.132771    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:42:55.228567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.222230    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-model-cache\") pod \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") "
Oct 30 14:42:55.228567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.222355    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tls-certs\") pod \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") "
Oct 30 14:42:55.228567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.222392    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-dshm\") pod \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") "
Oct 30 14:42:55.228567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.222613    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-home\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-home\") pod \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") "
Oct 30 14:42:55.228567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.222747    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb56386-09e1-4fa0-9a80-f291fc293fd1-kube-api-access-w6hk5\" (UniqueName: \"kubernetes.io/projected/bcb56386-09e1-4fa0-9a80-f291fc293fd1-kube-api-access-w6hk5\") pod \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") "
Oct 30 14:42:55.228567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.222806    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tmp-dir\") pod \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") "
Oct 30 14:42:55.238572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.234837    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tmp-dir" pod "bcb56386-09e1-4fa0-9a80-f291fc293fd1" (UID: "bcb56386-09e1-4fa0-9a80-f291fc293fd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:42:55.238572 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.235055    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-model-cache" pod "bcb56386-09e1-4fa0-9a80-f291fc293fd1" (UID: "bcb56386-09e1-4fa0-9a80-f291fc293fd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:42:55.257562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.244645    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-home" pod "bcb56386-09e1-4fa0-9a80-f291fc293fd1" (UID: "bcb56386-09e1-4fa0-9a80-f291fc293fd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:42:55.363621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.335824    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:42:55.363621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.335868    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:42:55.363621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.335884    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:42:55.469563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.461438    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tls-certs" pod "bcb56386-09e1-4fa0-9a80-f291fc293fd1" (UID: "bcb56386-09e1-4fa0-9a80-f291fc293fd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:42:55.470964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.470336    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tls-certs\") pod \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\" (UID: \"bcb56386-09e1-4fa0-9a80-f291fc293fd1\") "
Oct 30 14:42:55.470964 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:42:55.470686    4530 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/bcb56386-09e1-4fa0-9a80-f291fc293fd1/volumes/kubernetes.io~secret/tls-certs
Oct 30 14:42:55.470964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.470939    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tls-certs" pod "bcb56386-09e1-4fa0-9a80-f291fc293fd1" (UID: "bcb56386-09e1-4fa0-9a80-f291fc293fd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:42:55.474558 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.474517    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb56386-09e1-4fa0-9a80-f291fc293fd1-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:42:55.546595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.530947    4530 generic.go:356] "Generic (PLEG): container finished" podID="bcb56386-09e1-4fa0-9a80-f291fc293fd1" containerID="98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41" exitCode=0
Oct 30 14:42:55.547112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.539474    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"
Oct 30 14:42:55.547826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.539506    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2" event={"ID":"bcb56386-09e1-4fa0-9a80-f291fc293fd1","Type":"ContainerDied","Data":"98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41"}
Oct 30 14:42:55.548044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.548014    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2" event={"ID":"bcb56386-09e1-4fa0-9a80-f291fc293fd1","Type":"ContainerDied","Data":"f84752bc013d3f60ec1daf9ac843951bd5544ab49aa7c2488d2fea67386f5fc1"}
Oct 30 14:42:55.548196 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.548178    4530 scope.go:122] "RemoveContainer" containerID="98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41"
Oct 30 14:42:55.548416 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.539848    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 14:42:55.556027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.548574    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 14:42:55.579952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.579521    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-dshm" pod "bcb56386-09e1-4fa0-9a80-f291fc293fd1" (UID: "bcb56386-09e1-4fa0-9a80-f291fc293fd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:42:55.590453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.590415    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb56386-09e1-4fa0-9a80-f291fc293fd1-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:42:55.590786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.590760    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" event={"ID":"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d","Type":"ContainerStarted","Data":"7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f"}
Oct 30 14:42:55.621611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.618830    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb56386-09e1-4fa0-9a80-f291fc293fd1-kube-api-access-w6hk5" pod "bcb56386-09e1-4fa0-9a80-f291fc293fd1" (UID: "bcb56386-09e1-4fa0-9a80-f291fc293fd1"). InnerVolumeSpecName "kube-api-access-w6hk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:42:55.634039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.634005    4530 scope.go:122] "RemoveContainer" containerID="98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41"
Oct 30 14:42:55.634467 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:42:55.634440    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41\": container with ID starting with 98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41 not found: ID does not exist" containerID="98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41"
Oct 30 14:42:55.634604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.634480    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41"} err="failed to get container status \"98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41\": rpc error: code = NotFound desc = could not find container \"98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41\": container with ID starting with 98a079339b479286efa0967eab6ce2dd6e0762336b596a37db2cee4941277f41 not found: ID does not exist"
Oct 30 14:42:55.717570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.715636    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6hk5\" (UniqueName: \"kubernetes.io/projected/bcb56386-09e1-4fa0-9a80-f291fc293fd1-kube-api-access-w6hk5\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:42:55.841401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.839583    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:42:55.841401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.840194    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 14:42:55.841401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.840237    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 14:42:55.935752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.935641    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"]
Oct 30 14:42:55.941170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:55.941140    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2"]
Oct 30 14:42:56.031080 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:56.030667    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-c7c37b/auth-test-model-c7c37b-kserve-bfc8b9f7-98mb2" podUID="bcb56386-09e1-4fa0-9a80-f291fc293fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.1.26:8000/ready\": context deadline exceeded"
Oct 30 14:42:56.544663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:56.544519    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 14:42:56.544663 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:56.544609    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 14:42:56.840242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:56.840188    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 14:42:56.840463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:56.840264    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 14:42:57.255604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:57.254882    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb56386-09e1-4fa0-9a80-f291fc293fd1" path="/var/lib/kubelet/pods/bcb56386-09e1-4fa0-9a80-f291fc293fd1/volumes"
Oct 30 14:42:57.535626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:57.535502    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 14:42:57.535626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:57.535580    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 14:42:57.850819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:57.844650    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 14:42:57.850819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:57.846290    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 14:42:58.535386 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:58.535345    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 14:42:58.535630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:58.535403    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 14:42:58.839170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:58.839134    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 14:42:58.839453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:58.839428    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 14:42:59.546498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:59.546455    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 30 14:42:59.547051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:59.546523    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 30 14:42:59.839559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:59.839270    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 14:42:59.839559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:59.839325    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 14:42:59.980733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:42:59.980690    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz" podUID="b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc" containerName="main" containerID="cri-o://f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01" gracePeriod=45
Oct 30 14:43:00.355560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.353072    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:43:00.542124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.542096    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:43:00.552957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.552933    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:43:00.628286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.628240    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-kube-api-access-jph52\" (UniqueName: \"kubernetes.io/projected/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-kube-api-access-jph52\") pod \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") "
Oct 30 14:43:00.628516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.628337    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-dshm\") pod \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") "
Oct 30 14:43:00.628516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.628435    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tmp-dir\") pod \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") "
Oct 30 14:43:00.628516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.628471    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-home\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-home\") pod \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") "
Oct 30 14:43:00.628781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.628595    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-model-cache\") pod \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") "
Oct 30 14:43:00.628781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.628614    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tls-certs\") pod \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\" (UID: \"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc\") "
Oct 30 14:43:00.628901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.628830    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tmp-dir" pod "b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc" (UID: "b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:00.629012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.628838    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-home" pod "b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc" (UID: "b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:00.629012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.628979    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-model-cache" pod "b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc" (UID: "b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:00.629152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.629135    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:00.629205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.629158    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:00.629205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.629171    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:00.631037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.631000    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tls-certs" pod "b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc" (UID: "b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:43:00.631145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.631070    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-dshm" pod "b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc" (UID: "b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:00.631145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.631126    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-kube-api-access-jph52" pod "b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc" (UID: "b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc"). InnerVolumeSpecName "kube-api-access-jph52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:43:00.731078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.731040    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jph52\" (UniqueName: \"kubernetes.io/projected/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-kube-api-access-jph52\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:00.731078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.731073    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:00.731078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.731083    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:00.771930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.771837    4530 generic.go:356] "Generic (PLEG): container finished" podID="b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc" containerID="f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01" exitCode=0
Oct 30 14:43:00.771930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.771886    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"
Oct 30 14:43:00.771930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.771919    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz" event={"ID":"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc","Type":"ContainerDied","Data":"f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01"}
Oct 30 14:43:00.772202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.771962    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz" event={"ID":"b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc","Type":"ContainerDied","Data":"5865df31c647a1addb2f028d4fba6b5a7672858be7c20577680ae8aca969d21a"}
Oct 30 14:43:00.772202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.771981    4530 scope.go:122] "RemoveContainer" containerID="f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01"
Oct 30 14:43:00.773611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.773592    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:43:00.791638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.787906    4530 scope.go:122] "RemoveContainer" containerID="f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01"
Oct 30 14:43:00.791638 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:43:00.788937    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01\": container with ID starting with f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01 not found: ID does not exist" containerID="f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01"
Oct 30 14:43:00.791638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.788976    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01"} err="failed to get container status \"f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01\": rpc error: code = NotFound desc = could not find container \"f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01\": container with ID starting with f030c53dabdf00c3bb23618bf117637c68500a57d26d61048418f587fb980f01 not found: ID does not exist"
Oct 30 14:43:00.839429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.839392    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 14:43:00.839620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.839447    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 14:43:00.917660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.917615    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"]
Oct 30 14:43:00.921706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:00.921676    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e50e63/auth-test-model-e50e63-kserve-7c7dfcb957-vf9vz"]
Oct 30 14:43:01.118173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:01.118140    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc" path="/var/lib/kubelet/pods/b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc/volumes"
Oct 30 14:43:01.870385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:01.870340    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"]
Oct 30 14:43:01.870929 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:01.870675    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy" containerID="cri-o://9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093" gracePeriod=30
Oct 30 14:43:02.290260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:02.290171    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:43:02.290686 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:02.290667    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:43:02.292802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:02.292775    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:43:06.808566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.807597    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"]
Oct 30 14:43:06.809108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.808805    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1e5cbbcf-0058-44e5-8e80-2c4fb100bf70" containerName="maas-api"
Oct 30 14:43:06.809108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.808828    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5cbbcf-0058-44e5-8e80-2c4fb100bf70" containerName="maas-api"
Oct 30 14:43:06.809108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.808910    4530 cpu_manager.go:414] "containerMap: removing container" podUID="84c3cc86-1167-4863-8f71-2c66a873b96a" containerName="payload-processing"
Oct 30 14:43:06.809108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.808919    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c3cc86-1167-4863-8f71-2c66a873b96a" containerName="payload-processing"
Oct 30 14:43:06.809108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.808940    4530 cpu_manager.go:414] "containerMap: removing container" podUID="bcb56386-09e1-4fa0-9a80-f291fc293fd1" containerName="main"
Oct 30 14:43:06.809108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.808948    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb56386-09e1-4fa0-9a80-f291fc293fd1" containerName="main"
Oct 30 14:43:06.809108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.808974    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc" containerName="main"
Oct 30 14:43:06.809108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.808981    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc" containerName="main"
Oct 30 14:43:06.809532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.809160    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c3cc86-1167-4863-8f71-2c66a873b96a" containerName="payload-processing"
Oct 30 14:43:06.809532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.809180    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb56386-09e1-4fa0-9a80-f291fc293fd1" containerName="main"
Oct 30 14:43:06.809532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.809193    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ba4e6e-78fb-418f-a4c0-7b0f3d9871cc" containerName="main"
Oct 30 14:43:06.809532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.809206    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5cbbcf-0058-44e5-8e80-2c4fb100bf70" containerName="maas-api"
Oct 30 14:43:06.840678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.839834    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"
Oct 30 14:43:06.894899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.890752    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzmhk\" (UniqueName: \"kubernetes.io/projected/4b9dbd02-5964-42ef-a2e7-c15119066eb6-kube-api-access-jzmhk\") pod \"payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47\" (UID: \"4b9dbd02-5964-42ef-a2e7-c15119066eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"
Oct 30 14:43:06.894899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.890873    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9dbd02-5964-42ef-a2e7-c15119066eb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47\" (UID: \"4b9dbd02-5964-42ef-a2e7-c15119066eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"
Oct 30 14:43:06.908389 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.906364    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"]
Oct 30 14:43:06.954902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.954870    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a1854b\""
Oct 30 14:43:06.981863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.981825    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"]
Oct 30 14:43:06.989564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.988096    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"
Oct 30 14:43:06.991009 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.990940    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"]
Oct 30 14:43:06.993064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.993036    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzmhk\" (UniqueName: \"kubernetes.io/projected/4b9dbd02-5964-42ef-a2e7-c15119066eb6-kube-api-access-jzmhk\") pod \"payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47\" (UID: \"4b9dbd02-5964-42ef-a2e7-c15119066eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"
Oct 30 14:43:06.993212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.993144    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9dbd02-5964-42ef-a2e7-c15119066eb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47\" (UID: \"4b9dbd02-5964-42ef-a2e7-c15119066eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"
Oct 30 14:43:06.994314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:06.994285    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9dbd02-5964-42ef-a2e7-c15119066eb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47\" (UID: \"4b9dbd02-5964-42ef-a2e7-c15119066eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"
Oct 30 14:43:07.015100 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.015077    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a1854b-dockercfg-6c7cw\""
Oct 30 14:43:07.025233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.025173    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzmhk\" (UniqueName: \"kubernetes.io/projected/4b9dbd02-5964-42ef-a2e7-c15119066eb6-kube-api-access-jzmhk\") pod \"payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47\" (UID: \"4b9dbd02-5964-42ef-a2e7-c15119066eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"
Oct 30 14:43:07.094984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.094942    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1a6fcb-ea05-4886-a998-df6cce7f3157-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd\" (UID: \"9c1a6fcb-ea05-4886-a998-df6cce7f3157\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"
Oct 30 14:43:07.095364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.095341    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4mp8\" (UniqueName: \"kubernetes.io/projected/9c1a6fcb-ea05-4886-a998-df6cce7f3157-kube-api-access-d4mp8\") pod \"payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd\" (UID: \"9c1a6fcb-ea05-4886-a998-df6cce7f3157\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"
Oct 30 14:43:07.178364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.178339    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:43:07.196426 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.196385    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4mp8\" (UniqueName: \"kubernetes.io/projected/9c1a6fcb-ea05-4886-a998-df6cce7f3157-kube-api-access-d4mp8\") pod \"payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd\" (UID: \"9c1a6fcb-ea05-4886-a998-df6cce7f3157\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"
Oct 30 14:43:07.196661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.196633    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1a6fcb-ea05-4886-a998-df6cce7f3157-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd\" (UID: \"9c1a6fcb-ea05-4886-a998-df6cce7f3157\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"
Oct 30 14:43:07.197372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.197347    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1a6fcb-ea05-4886-a998-df6cce7f3157-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd\" (UID: \"9c1a6fcb-ea05-4886-a998-df6cce7f3157\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"
Oct 30 14:43:07.223357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.223336    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4mp8\" (UniqueName: \"kubernetes.io/projected/9c1a6fcb-ea05-4886-a998-df6cce7f3157-kube-api-access-d4mp8\") pod \"payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd\" (UID: \"9c1a6fcb-ea05-4886-a998-df6cce7f3157\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"
Oct 30 14:43:07.245688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.245655    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"
Oct 30 14:43:07.297951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.297915    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istiod-ca-cert\") pod \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") "
Oct 30 14:43:07.298121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.297960    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-certs\") pod \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") "
Oct 30 14:43:07.298121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.298028    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-podinfo\") pod \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") "
Oct 30 14:43:07.298121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.298102    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-data\") pod \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") "
Oct 30 14:43:07.298304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.298152    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-credential-socket\") pod \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") "
Oct 30 14:43:07.298304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.298249    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-kube-api-access-dxlf8\" (UniqueName: \"kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-kube-api-access-dxlf8\") pod \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") "
Oct 30 14:43:07.298304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.298278    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-token\" (UniqueName: \"kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-token\") pod \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") "
Oct 30 14:43:07.298475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.298343    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-envoy\") pod \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") "
Oct 30 14:43:07.298475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.298361    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-certs" pod "330c4039-1ce9-4eee-af89-a18cbc37e9a0" (UID: "330c4039-1ce9-4eee-af89-a18cbc37e9a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:07.298475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.298401    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-socket\") pod \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\" (UID: \"330c4039-1ce9-4eee-af89-a18cbc37e9a0\") "
Oct 30 14:43:07.298475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.298446    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-data" pod "330c4039-1ce9-4eee-af89-a18cbc37e9a0" (UID: "330c4039-1ce9-4eee-af89-a18cbc37e9a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:07.298754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.298495    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istiod-ca-cert" pod "330c4039-1ce9-4eee-af89-a18cbc37e9a0" (UID: "330c4039-1ce9-4eee-af89-a18cbc37e9a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:43:07.298754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.298714    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-credential-socket" pod "330c4039-1ce9-4eee-af89-a18cbc37e9a0" (UID: "330c4039-1ce9-4eee-af89-a18cbc37e9a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:07.298968 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.298945    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-socket" pod "330c4039-1ce9-4eee-af89-a18cbc37e9a0" (UID: "330c4039-1ce9-4eee-af89-a18cbc37e9a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:07.299255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.299231    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:07.299374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.299259    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:07.299374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.299274    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:07.299374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.299288    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:07.299374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.299300    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:07.300660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.300632    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-kube-api-access-dxlf8" pod "330c4039-1ce9-4eee-af89-a18cbc37e9a0" (UID: "330c4039-1ce9-4eee-af89-a18cbc37e9a0"). InnerVolumeSpecName "kube-api-access-dxlf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:43:07.300660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.300639    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-envoy" pod "330c4039-1ce9-4eee-af89-a18cbc37e9a0" (UID: "330c4039-1ce9-4eee-af89-a18cbc37e9a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:07.300822 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.300730    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-podinfo" pod "330c4039-1ce9-4eee-af89-a18cbc37e9a0" (UID: "330c4039-1ce9-4eee-af89-a18cbc37e9a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:43:07.301028 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.301013    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-token" pod "330c4039-1ce9-4eee-af89-a18cbc37e9a0" (UID: "330c4039-1ce9-4eee-af89-a18cbc37e9a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:43:07.316374 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.316345    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"
Oct 30 14:43:07.401707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.401670    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:07.401853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.401711    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxlf8\" (UniqueName: \"kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-kube-api-access-dxlf8\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:07.401853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.401729    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:07.401853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.401744    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330c4039-1ce9-4eee-af89-a18cbc37e9a0-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:07.563488 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:43:07.563457    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9dbd02_5964_42ef_a2e7_c15119066eb6.slice/crio-bf18239b2c1e874fd44d3a3cfc3185d72d30571a2d769801709cfb71dc6b4b67 WatchSource:0}: Error finding container bf18239b2c1e874fd44d3a3cfc3185d72d30571a2d769801709cfb71dc6b4b67: Status 404 returned error can't find the container with id bf18239b2c1e874fd44d3a3cfc3185d72d30571a2d769801709cfb71dc6b4b67
Oct 30 14:43:07.565037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.565007    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"]
Oct 30 14:43:07.641996 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:43:07.641967    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1a6fcb_ea05_4886_a998_df6cce7f3157.slice/crio-e5c4ac1dce862a18e0e36ac1ead1f2e85e0f67e999cbf9827ae818b20c166c7c WatchSource:0}: Error finding container e5c4ac1dce862a18e0e36ac1ead1f2e85e0f67e999cbf9827ae818b20c166c7c: Status 404 returned error can't find the container with id e5c4ac1dce862a18e0e36ac1ead1f2e85e0f67e999cbf9827ae818b20c166c7c
Oct 30 14:43:07.644187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.644156    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"]
Oct 30 14:43:07.948121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.948032    4530 generic.go:356] "Generic (PLEG): container finished" podID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerID="9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093" exitCode=0
Oct 30 14:43:07.948121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.948089    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" event={"ID":"330c4039-1ce9-4eee-af89-a18cbc37e9a0","Type":"ContainerDied","Data":"9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093"}
Oct 30 14:43:07.948721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.948102    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"
Oct 30 14:43:07.948721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.948158    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx" event={"ID":"330c4039-1ce9-4eee-af89-a18cbc37e9a0","Type":"ContainerDied","Data":"da15264a53ca7280d59f8053a64a6c20f3c87bf64a0ad731e915b5a342b98640"}
Oct 30 14:43:07.948721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.948191    4530 scope.go:122] "RemoveContainer" containerID="9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093"
Oct 30 14:43:07.951211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.950988    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd" event={"ID":"9c1a6fcb-ea05-4886-a998-df6cce7f3157","Type":"ContainerStarted","Data":"8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac"}
Oct 30 14:43:07.951211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.951041    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd" event={"ID":"9c1a6fcb-ea05-4886-a998-df6cce7f3157","Type":"ContainerStarted","Data":"e5c4ac1dce862a18e0e36ac1ead1f2e85e0f67e999cbf9827ae818b20c166c7c"}
Oct 30 14:43:07.951383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.951215    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"
Oct 30 14:43:07.955655 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.955630    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47" event={"ID":"4b9dbd02-5964-42ef-a2e7-c15119066eb6","Type":"ContainerStarted","Data":"a75eac50009385dabdabf28b8a2db62a1df7f932f7fbbc011a405c81596f6624"}
Oct 30 14:43:07.955788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.955676    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47" event={"ID":"4b9dbd02-5964-42ef-a2e7-c15119066eb6","Type":"ContainerStarted","Data":"bf18239b2c1e874fd44d3a3cfc3185d72d30571a2d769801709cfb71dc6b4b67"}
Oct 30 14:43:07.955788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.955710    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"
Oct 30 14:43:07.965438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.965418    4530 scope.go:122] "RemoveContainer" containerID="631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0"
Oct 30 14:43:07.976809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.976786    4530 scope.go:122] "RemoveContainer" containerID="9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093"
Oct 30 14:43:07.977064 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:43:07.977037    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093\": container with ID starting with 9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093 not found: ID does not exist" containerID="9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093"
Oct 30 14:43:07.977124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.977072    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093"} err="failed to get container status \"9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093\": rpc error: code = NotFound desc = could not find container \"9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093\": container with ID starting with 9259f11c3c4311319484c9deeca0555d51722051096a5306081a6a585ccf5093 not found: ID does not exist"
Oct 30 14:43:07.977124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.977090    4530 scope.go:122] "RemoveContainer" containerID="631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0"
Oct 30 14:43:07.977309 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:43:07.977289    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0\": container with ID starting with 631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0 not found: ID does not exist" containerID="631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0"
Oct 30 14:43:07.977355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:07.977314    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0"} err="failed to get container status \"631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0\": rpc error: code = NotFound desc = could not find container \"631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0\": container with ID starting with 631e98a0e9ee5de178f4919318f59e03bcfe51321ce456466d861cf97d8e55f0 not found: ID does not exist"
Oct 30 14:43:08.013926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:08.013900    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"]
Oct 30 14:43:08.019866 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:08.019823    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d3a829-openshift-default-74d65df7df-q2hjx"]
Oct 30 14:43:08.062055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:08.062007    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd" podStartSLOduration=2.061994021 podStartE2EDuration="2.061994021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:08.061053224 +0000 UTC m=+3253.092952164" watchObservedRunningTime="2026-10-30 14:43:08.061994021 +0000 UTC m=+3253.093892960"
Oct 30 14:43:08.106733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:08.106688    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47" podStartSLOduration=2.106672594 podStartE2EDuration="2.106672594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:08.106298068 +0000 UTC m=+3253.138197009" watchObservedRunningTime="2026-10-30 14:43:08.106672594 +0000 UTC m=+3253.138571535"
Oct 30 14:43:09.118185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:09.118149    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" path="/var/lib/kubelet/pods/330c4039-1ce9-4eee-af89-a18cbc37e9a0/volumes"
Oct 30 14:43:11.368410 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:43:11.368371    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:43:13.446649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.446611    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"]
Oct 30 14:43:13.448418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.448387    4530 cpu_manager.go:414] "containerMap: removing container" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy"
Oct 30 14:43:13.448688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.448670    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy"
Oct 30 14:43:13.448852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.448838    4530 cpu_manager.go:414] "containerMap: removing container" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy"
Oct 30 14:43:13.452598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.452572    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy"
Oct 30 14:43:13.452956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.452936    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy"
Oct 30 14:43:13.459640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.459615    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="330c4039-1ce9-4eee-af89-a18cbc37e9a0" containerName="istio-proxy"
Oct 30 14:43:13.495010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.494978    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"]
Oct 30 14:43:13.495530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.495452    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"
Oct 30 14:43:13.519859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.517844    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6176c3\""
Oct 30 14:43:13.519859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.518590    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6176c3-dockercfg-2dpjq\""
Oct 30 14:43:13.537567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.530990    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx\" (UID: \"05d6ddc5-2dd7-4148-be66-b0b0ae909d37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"
Oct 30 14:43:13.538144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.538114    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmlrw\" (UniqueName: \"kubernetes.io/projected/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-kube-api-access-zmlrw\") pod \"payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx\" (UID: \"05d6ddc5-2dd7-4148-be66-b0b0ae909d37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"
Oct 30 14:43:13.566993 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.566963    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"]
Oct 30 14:43:13.600229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.600197    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"
Oct 30 14:43:13.604784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.603511    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"]
Oct 30 14:43:13.643574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.643527    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx\" (UID: \"05d6ddc5-2dd7-4148-be66-b0b0ae909d37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"
Oct 30 14:43:13.643743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.643710    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxpqh\" (UniqueName: \"kubernetes.io/projected/7aa3d073-f26b-49a3-b32e-25dea19ddf12-kube-api-access-mxpqh\") pod \"payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2\" (UID: \"7aa3d073-f26b-49a3-b32e-25dea19ddf12\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"
Oct 30 14:43:13.643786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.643744    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmlrw\" (UniqueName: \"kubernetes.io/projected/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-kube-api-access-zmlrw\") pod \"payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx\" (UID: \"05d6ddc5-2dd7-4148-be66-b0b0ae909d37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"
Oct 30 14:43:13.643836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.643822    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa3d073-f26b-49a3-b32e-25dea19ddf12-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2\" (UID: \"7aa3d073-f26b-49a3-b32e-25dea19ddf12\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"
Oct 30 14:43:13.644233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.644213    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx\" (UID: \"05d6ddc5-2dd7-4148-be66-b0b0ae909d37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"
Oct 30 14:43:13.671679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.671651    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmlrw\" (UniqueName: \"kubernetes.io/projected/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-kube-api-access-zmlrw\") pod \"payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx\" (UID: \"05d6ddc5-2dd7-4148-be66-b0b0ae909d37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"
Oct 30 14:43:13.756619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.756473    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxpqh\" (UniqueName: \"kubernetes.io/projected/7aa3d073-f26b-49a3-b32e-25dea19ddf12-kube-api-access-mxpqh\") pod \"payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2\" (UID: \"7aa3d073-f26b-49a3-b32e-25dea19ddf12\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"
Oct 30 14:43:13.756807 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.756631    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa3d073-f26b-49a3-b32e-25dea19ddf12-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2\" (UID: \"7aa3d073-f26b-49a3-b32e-25dea19ddf12\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"
Oct 30 14:43:13.757723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.757692    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa3d073-f26b-49a3-b32e-25dea19ddf12-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2\" (UID: \"7aa3d073-f26b-49a3-b32e-25dea19ddf12\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"
Oct 30 14:43:13.785283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.785248    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxpqh\" (UniqueName: \"kubernetes.io/projected/7aa3d073-f26b-49a3-b32e-25dea19ddf12-kube-api-access-mxpqh\") pod \"payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2\" (UID: \"7aa3d073-f26b-49a3-b32e-25dea19ddf12\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"
Oct 30 14:43:13.832064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.832034    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"
Oct 30 14:43:13.922244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:13.921932    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"
Oct 30 14:43:14.143253 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:43:14.143227    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d6ddc5_2dd7_4148_be66_b0b0ae909d37.slice/crio-f5789ba8706647627a7e0a9f9212537ebfb3796bc7547c2f0237f6aa779b8e8d WatchSource:0}: Error finding container f5789ba8706647627a7e0a9f9212537ebfb3796bc7547c2f0237f6aa779b8e8d: Status 404 returned error can't find the container with id f5789ba8706647627a7e0a9f9212537ebfb3796bc7547c2f0237f6aa779b8e8d
Oct 30 14:43:14.145760 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:14.145734    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"]
Oct 30 14:43:14.238231 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:43:14.238204    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa3d073_f26b_49a3_b32e_25dea19ddf12.slice/crio-7231b92ad5de96f9a4c0b97b57ad583aa69aee47c2969108ffe94b03c1cb823c WatchSource:0}: Error finding container 7231b92ad5de96f9a4c0b97b57ad583aa69aee47c2969108ffe94b03c1cb823c: Status 404 returned error can't find the container with id 7231b92ad5de96f9a4c0b97b57ad583aa69aee47c2969108ffe94b03c1cb823c
Oct 30 14:43:14.239138 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:14.239113    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"]
Oct 30 14:43:15.021284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:15.021242    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx" event={"ID":"05d6ddc5-2dd7-4148-be66-b0b0ae909d37","Type":"ContainerStarted","Data":"d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053"}
Oct 30 14:43:15.021786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:15.021312    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx" event={"ID":"05d6ddc5-2dd7-4148-be66-b0b0ae909d37","Type":"ContainerStarted","Data":"f5789ba8706647627a7e0a9f9212537ebfb3796bc7547c2f0237f6aa779b8e8d"}
Oct 30 14:43:15.021786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:15.021477    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"
Oct 30 14:43:15.023112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:15.023082    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2" event={"ID":"7aa3d073-f26b-49a3-b32e-25dea19ddf12","Type":"ContainerStarted","Data":"90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197"}
Oct 30 14:43:15.023236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:15.023140    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2" event={"ID":"7aa3d073-f26b-49a3-b32e-25dea19ddf12","Type":"ContainerStarted","Data":"7231b92ad5de96f9a4c0b97b57ad583aa69aee47c2969108ffe94b03c1cb823c"}
Oct 30 14:43:15.023236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:15.023169    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"
Oct 30 14:43:15.067237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:15.067183    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx" podStartSLOduration=2.067168833 podStartE2EDuration="2.067168833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:15.067037015 +0000 UTC m=+3260.098935945" watchObservedRunningTime="2026-10-30 14:43:15.067168833 +0000 UTC m=+3260.099067773"
Oct 30 14:43:15.115808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:15.115756    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2" podStartSLOduration=2.115738826 podStartE2EDuration="2.115738826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:15.115173343 +0000 UTC m=+3260.147072281" watchObservedRunningTime="2026-10-30 14:43:15.115738826 +0000 UTC m=+3260.147637766"
Oct 30 14:43:18.968732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:18.968697    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"
Oct 30 14:43:18.969236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:18.969010    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"
Oct 30 14:43:19.631256 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:19.631219    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"]
Oct 30 14:43:19.631724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:19.631688    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq" podUID="f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d" containerName="maas-api" containerID="cri-o://c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907" gracePeriod=30
Oct 30 14:43:20.003872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.003821    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:43:20.220554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.216046    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-metrics-certs\") pod \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") "
Oct 30 14:43:20.220937 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.220905    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-kube-api-access-lvm9q\" (UniqueName: \"kubernetes.io/projected/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-kube-api-access-lvm9q\") pod \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") "
Oct 30 14:43:20.221245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.221224    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-maas-api-tls\") pod \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") "
Oct 30 14:43:20.221557 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.221520    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-openshift-service-ca\") pod \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\" (UID: \"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d\") "
Oct 30 14:43:20.263283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.263246    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-openshift-service-ca" pod "f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d" (UID: "f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:43:20.283596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.267719    4530 generic.go:356] "Generic (PLEG): container finished" podID="f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d" containerID="c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907" exitCode=0
Oct 30 14:43:20.283596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.267802    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq" event={"ID":"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d","Type":"ContainerDied","Data":"c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907"}
Oct 30 14:43:20.283596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.267874    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq" event={"ID":"f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d","Type":"ContainerDied","Data":"fd2b6d836c41152a76d4c06f79c980ca2d1127008281f63fd36daa08b65dea2e"}
Oct 30 14:43:20.283596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.277128    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-kube-api-access-lvm9q" pod "f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d" (UID: "f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d"). InnerVolumeSpecName "kube-api-access-lvm9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:43:20.283596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.281801    4530 scope.go:122] "RemoveContainer" containerID="c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907"
Oct 30 14:43:20.283596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.282052    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"
Oct 30 14:43:20.284802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.284730    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-maas-api-tls" pod "f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d" (UID: "f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:43:20.290137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.290005    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-metrics-certs" pod "f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d" (UID: "f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:43:20.320461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.320375    4530 scope.go:122] "RemoveContainer" containerID="c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907"
Oct 30 14:43:20.321394 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:43:20.321364    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907\": container with ID starting with c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907 not found: ID does not exist" containerID="c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907"
Oct 30 14:43:20.321569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.321403    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907"} err="failed to get container status \"c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907\": rpc error: code = NotFound desc = could not find container \"c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907\": container with ID starting with c6bd8a626cf0a9f573bebed74f1978a0a7748ccdf483807b3db9354c4a5f7907 not found: ID does not exist"
Oct 30 14:43:20.323682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.323658    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:20.323772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.323687    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvm9q\" (UniqueName: \"kubernetes.io/projected/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-kube-api-access-lvm9q\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:20.323772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.323702    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:20.323772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.323715    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:20.664566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.660762    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"]
Oct 30 14:43:20.684702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:20.683960    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c7c37b-65db49b8b5-zl7tq"]
Oct 30 14:43:21.116738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:21.116707    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d" path="/var/lib/kubelet/pods/f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d/volumes"
Oct 30 14:43:24.547269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.547233    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:43:24.602630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.602586    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"]
Oct 30 14:43:24.603749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.603723    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d" containerName="maas-api"
Oct 30 14:43:24.603749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.603747    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d" containerName="maas-api"
Oct 30 14:43:24.604002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.603985    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6dd8280-f315-4321-a5ab-fd2f1a8a9b6d" containerName="maas-api"
Oct 30 14:43:24.610284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.610233    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.629621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.629594    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"]
Oct 30 14:43:24.631135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.631112    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4be240-openshift-default-dockercfg-nhtzs\""
Oct 30 14:43:24.807667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.789321    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-socket\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.808023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.807984    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-token\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.808274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.808253    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istiod-ca-cert\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.808419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.808398    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-envoy\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.821569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.808744    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-credential-socket\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.821767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.821723    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-podinfo\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.821841 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.821826    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb4qt\" (UniqueName: \"kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-kube-api-access-qb4qt\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.841566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.821896    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-data\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.841566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.821962    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-certs\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.923563    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb4qt\" (UniqueName: \"kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-kube-api-access-qb4qt\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.923660    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-data\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.923717    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-certs\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.923784    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-socket\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.923829    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-token\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.923888    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istiod-ca-cert\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.923916    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-envoy\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.924097    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-credential-socket\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.924146    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-podinfo\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.925084    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-socket\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.926018    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-data\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.926249    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-certs\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.926857    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istiod-ca-cert\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.929562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.928843    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-credential-socket\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.964432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.964397    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-envoy\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.973643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.969158    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-podinfo\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.980011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.979977    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-token\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:24.985624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:24.982332    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb4qt\" (UniqueName: \"kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-kube-api-access-qb4qt\") pod \"e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") " pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:25.236868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:25.236776    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:25.564333 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:43:25.564302    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17140e84_9ba9_4ecc_83cf_9498e2ccd8b4.slice/crio-6758f07759eaf05980aaa55e1f90460b4d8331fe395ecd8c28e35070ef44c173 WatchSource:0}: Error finding container 6758f07759eaf05980aaa55e1f90460b4d8331fe395ecd8c28e35070ef44c173: Status 404 returned error can't find the container with id 6758f07759eaf05980aaa55e1f90460b4d8331fe395ecd8c28e35070ef44c173
Oct 30 14:43:25.565241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:25.565214    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"]
Oct 30 14:43:25.567489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:25.567451    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:43:25.567591 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:25.567523    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:43:25.567591 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:25.567575    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:43:26.033979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:26.033825    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"
Oct 30 14:43:26.033979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:26.033927    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"
Oct 30 14:43:26.343380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:26.343344    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" event={"ID":"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4","Type":"ContainerStarted","Data":"2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9"}
Oct 30 14:43:26.343577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:26.343395    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" event={"ID":"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4","Type":"ContainerStarted","Data":"6758f07759eaf05980aaa55e1f90460b4d8331fe395ecd8c28e35070ef44c173"}
Oct 30 14:43:26.396821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:26.396773    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" podStartSLOduration=2.396759638 podStartE2EDuration="2.396759638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:26.394899258 +0000 UTC m=+3271.426798212" watchObservedRunningTime="2026-10-30 14:43:26.396759638 +0000 UTC m=+3271.428658642"
Oct 30 14:43:27.237119 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:27.237082    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:27.244185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:27.244146    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:27.351776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:27.351744    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:27.352818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:27.352795    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:43:28.392506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:28.392467    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"]
Oct 30 14:43:28.393062 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:28.392775    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n" podUID="61dcc1f8-e09e-463f-8c94-f8453bfdcf1f" containerName="payload-pre-processing" containerID="cri-o://7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83" gracePeriod=30
Oct 30 14:43:28.399930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:28.399895    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n" podUID="61dcc1f8-e09e-463f-8c94-f8453bfdcf1f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.1.34:9004: connect: connection refused"
Oct 30 14:43:28.723576 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:28.723509    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"
Oct 30 14:43:28.868562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:28.868411    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-plugins-config-volume" pod "61dcc1f8-e09e-463f-8c94-f8453bfdcf1f" (UID: "61dcc1f8-e09e-463f-8c94-f8453bfdcf1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:43:28.868562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:28.868493    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-plugins-config-volume\") pod \"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f\" (UID: \"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f\") "
Oct 30 14:43:28.920596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:28.916618    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-kube-api-access-pwn57\" (UniqueName: \"kubernetes.io/projected/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-kube-api-access-pwn57\") pod \"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f\" (UID: \"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f\") "
Oct 30 14:43:28.920596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:28.917487    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:29.046565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.045870    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-kube-api-access-pwn57" pod "61dcc1f8-e09e-463f-8c94-f8453bfdcf1f" (UID: "61dcc1f8-e09e-463f-8c94-f8453bfdcf1f"). InnerVolumeSpecName "kube-api-access-pwn57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:43:29.121413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.121379    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwn57\" (UniqueName: \"kubernetes.io/projected/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f-kube-api-access-pwn57\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:29.385886 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.385850    4530 generic.go:356] "Generic (PLEG): container finished" podID="61dcc1f8-e09e-463f-8c94-f8453bfdcf1f" containerID="7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83" exitCode=0
Oct 30 14:43:29.386075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.385900    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"
Oct 30 14:43:29.386075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.385936    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n" event={"ID":"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f","Type":"ContainerDied","Data":"7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83"}
Oct 30 14:43:29.386075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.385996    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n" event={"ID":"61dcc1f8-e09e-463f-8c94-f8453bfdcf1f","Type":"ContainerDied","Data":"125ff0ed4e291deb7abd8321c9f7d46a991ad8e3f7118e8ac1e55a4fa9b4e279"}
Oct 30 14:43:29.386075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.386019    4530 scope.go:122] "RemoveContainer" containerID="7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83"
Oct 30 14:43:29.399257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.399040    4530 scope.go:122] "RemoveContainer" containerID="7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83"
Oct 30 14:43:29.399503 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:43:29.399313    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83\": container with ID starting with 7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83 not found: ID does not exist" containerID="7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83"
Oct 30 14:43:29.399503 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.399342    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83"} err="failed to get container status \"7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83\": rpc error: code = NotFound desc = could not find container \"7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83\": container with ID starting with 7dc0dccbf150127855c0e12b51de7f58f31b22759ae0f24f6d9386519e952a83 not found: ID does not exist"
Oct 30 14:43:29.453746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.453718    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"]
Oct 30 14:43:29.461062 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.461035    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-6b4dbb5955-std4n"]
Oct 30 14:43:29.538043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.538009    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"]
Oct 30 14:43:29.538259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:29.538231    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy" containerID="cri-o://0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68" gracePeriod=30
Oct 30 14:43:31.226858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:31.226815    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61dcc1f8-e09e-463f-8c94-f8453bfdcf1f" path="/var/lib/kubelet/pods/61dcc1f8-e09e-463f-8c94-f8453bfdcf1f/volumes"
Oct 30 14:43:34.934365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:34.934330    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"]
Oct 30 14:43:34.934792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:34.934577    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy" containerID="cri-o://036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea" gracePeriod=30
Oct 30 14:43:34.955652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:34.955622    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:43:35.066408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.066373    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-podinfo\") pod \"b2abb44c-4a6d-44b2-8081-d24b067dda00\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") "
Oct 30 14:43:35.066613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.066429    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2abb44c-4a6d-44b2-8081-d24b067dda00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2abb44c-4a6d-44b2-8081-d24b067dda00-istiod-ca-cert\") pod \"b2abb44c-4a6d-44b2-8081-d24b067dda00\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") "
Oct 30 14:43:35.066613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.066465    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-data\") pod \"b2abb44c-4a6d-44b2-8081-d24b067dda00\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") "
Oct 30 14:43:35.066613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.066523    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-credential-socket\") pod \"b2abb44c-4a6d-44b2-8081-d24b067dda00\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") "
Oct 30 14:43:35.066835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.066634    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-socket\") pod \"b2abb44c-4a6d-44b2-8081-d24b067dda00\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") "
Oct 30 14:43:35.066835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.066691    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-certs\") pod \"b2abb44c-4a6d-44b2-8081-d24b067dda00\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") "
Oct 30 14:43:35.066835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.066780    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-envoy\") pod \"b2abb44c-4a6d-44b2-8081-d24b067dda00\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") "
Oct 30 14:43:35.067010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.066844    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-kube-api-access-5l2pq\" (UniqueName: \"kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-kube-api-access-5l2pq\") pod \"b2abb44c-4a6d-44b2-8081-d24b067dda00\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") "
Oct 30 14:43:35.067010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.066922    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-token\" (UniqueName: \"kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-token\") pod \"b2abb44c-4a6d-44b2-8081-d24b067dda00\" (UID: \"b2abb44c-4a6d-44b2-8081-d24b067dda00\") "
Oct 30 14:43:35.067010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.066957    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2abb44c-4a6d-44b2-8081-d24b067dda00-istiod-ca-cert" pod "b2abb44c-4a6d-44b2-8081-d24b067dda00" (UID: "b2abb44c-4a6d-44b2-8081-d24b067dda00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:43:35.067173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.067049    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-socket" pod "b2abb44c-4a6d-44b2-8081-d24b067dda00" (UID: "b2abb44c-4a6d-44b2-8081-d24b067dda00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:35.067264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.067245    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-data" pod "b2abb44c-4a6d-44b2-8081-d24b067dda00" (UID: "b2abb44c-4a6d-44b2-8081-d24b067dda00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:35.067428 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.067408    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-credential-socket" pod "b2abb44c-4a6d-44b2-8081-d24b067dda00" (UID: "b2abb44c-4a6d-44b2-8081-d24b067dda00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:35.067788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.067763    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2abb44c-4a6d-44b2-8081-d24b067dda00-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:35.067788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.067786    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:35.067970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.067795    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:35.067970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.067808    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:35.067970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.067902    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-certs" pod "b2abb44c-4a6d-44b2-8081-d24b067dda00" (UID: "b2abb44c-4a6d-44b2-8081-d24b067dda00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:35.070085 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.070048    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-token" pod "b2abb44c-4a6d-44b2-8081-d24b067dda00" (UID: "b2abb44c-4a6d-44b2-8081-d24b067dda00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:43:35.070671 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.070632    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-podinfo" pod "b2abb44c-4a6d-44b2-8081-d24b067dda00" (UID: "b2abb44c-4a6d-44b2-8081-d24b067dda00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:43:35.070967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.070930    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-envoy" pod "b2abb44c-4a6d-44b2-8081-d24b067dda00" (UID: "b2abb44c-4a6d-44b2-8081-d24b067dda00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:35.072881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.072847    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-kube-api-access-5l2pq" pod "b2abb44c-4a6d-44b2-8081-d24b067dda00" (UID: "b2abb44c-4a6d-44b2-8081-d24b067dda00"). InnerVolumeSpecName "kube-api-access-5l2pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:43:35.169516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.169483    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:35.169516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.169515    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:35.169731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.169525    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l2pq\" (UniqueName: \"kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-kube-api-access-5l2pq\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:35.169731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.169534    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:35.169731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.169560    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2abb44c-4a6d-44b2-8081-d24b067dda00-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:35.463737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.463644    4530 generic.go:356] "Generic (PLEG): container finished" podID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerID="0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68" exitCode=0
Oct 30 14:43:35.463737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.463713    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"
Oct 30 14:43:35.463737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.463720    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" event={"ID":"b2abb44c-4a6d-44b2-8081-d24b067dda00","Type":"ContainerDied","Data":"0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68"}
Oct 30 14:43:35.463978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.463799    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl" event={"ID":"b2abb44c-4a6d-44b2-8081-d24b067dda00","Type":"ContainerDied","Data":"c6d113bc7bdbfef329d30d4cd47f782fc3f516f8f2d13d2b34f4f1d43e5ad957"}
Oct 30 14:43:35.463978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.463821    4530 scope.go:122] "RemoveContainer" containerID="0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68"
Oct 30 14:43:35.474829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.474801    4530 scope.go:122] "RemoveContainer" containerID="184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472"
Oct 30 14:43:35.484965 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.484943    4530 scope.go:122] "RemoveContainer" containerID="0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68"
Oct 30 14:43:35.485249 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:43:35.485222    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68\": container with ID starting with 0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68 not found: ID does not exist" containerID="0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68"
Oct 30 14:43:35.485349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.485260    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68"} err="failed to get container status \"0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68\": rpc error: code = NotFound desc = could not find container \"0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68\": container with ID starting with 0e758741035828763ad5e7d76bf9cc538aab28534d8a4ff2bc050a93e6faeb68 not found: ID does not exist"
Oct 30 14:43:35.485349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.485282    4530 scope.go:122] "RemoveContainer" containerID="184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472"
Oct 30 14:43:35.485532 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:43:35.485512    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472\": container with ID starting with 184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472 not found: ID does not exist" containerID="184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472"
Oct 30 14:43:35.485617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.485571    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472"} err="failed to get container status \"184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472\": rpc error: code = NotFound desc = could not find container \"184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472\": container with ID starting with 184a8ba1cffd4a142f34a5720ca55d73a34b689b4c2404bcbf7bbefe393a6472 not found: ID does not exist"
Oct 30 14:43:35.528123 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.528092    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"]
Oct 30 14:43:35.542011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.541979    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-lrnfl"]
Oct 30 14:43:35.689948 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.689914    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"]
Oct 30 14:43:35.690681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.690658    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy"
Oct 30 14:43:35.690681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.690677    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy"
Oct 30 14:43:35.690857 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.690714    4530 cpu_manager.go:414] "containerMap: removing container" podUID="61dcc1f8-e09e-463f-8c94-f8453bfdcf1f" containerName="payload-pre-processing"
Oct 30 14:43:35.690857 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.690720    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61dcc1f8-e09e-463f-8c94-f8453bfdcf1f" containerName="payload-pre-processing"
Oct 30 14:43:35.690927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.690860    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy"
Oct 30 14:43:35.690927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.690871    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="61dcc1f8-e09e-463f-8c94-f8453bfdcf1f" containerName="payload-pre-processing"
Oct 30 14:43:35.696002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.695985    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:35.714841 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.714778    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d05e21-serving-cert\""
Oct 30 14:43:35.715257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.715231    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"]
Oct 30 14:43:35.902138 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.902091    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-metrics-certs\") pod \"maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:35.902771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.902738    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:35.903025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.903001    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp599\" (UniqueName: \"kubernetes.io/projected/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-kube-api-access-tp599\") pod \"maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:35.903300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:35.903281    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:36.004503 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:36.004409    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-metrics-certs\") pod \"maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:36.005046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:36.004609    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:36.005046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:36.004695    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp599\" (UniqueName: \"kubernetes.io/projected/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-kube-api-access-tp599\") pod \"maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:36.005046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:36.004722    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:36.005368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:36.005350    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:36.007231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:36.007199    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-metrics-certs\") pod \"maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:36.007360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:36.007207    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:36.031448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:36.031423    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp599\" (UniqueName: \"kubernetes.io/projected/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-kube-api-access-tp599\") pod \"maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:36.313798 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:36.313760    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:36.627008 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:43:36.626978    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a2193e_90af_42cd_8af9_6cdcdaeeb7e9.slice/crio-9237d4d881836719668103fdb14453400e68ca740475895d96f68cc4f7b9c43a WatchSource:0}: Error finding container 9237d4d881836719668103fdb14453400e68ca740475895d96f68cc4f7b9c43a: Status 404 returned error can't find the container with id 9237d4d881836719668103fdb14453400e68ca740475895d96f68cc4f7b9c43a
Oct 30 14:43:36.628382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:36.628355    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"]
Oct 30 14:43:37.117653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:37.117623    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" path="/var/lib/kubelet/pods/b2abb44c-4a6d-44b2-8081-d24b067dda00/volumes"
Oct 30 14:43:37.352355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:37.352270    4530 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:43:37.352355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:37.352329    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:43:37.512741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:37.512704    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7" event={"ID":"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9","Type":"ContainerStarted","Data":"91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd"}
Oct 30 14:43:37.512971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:37.512768    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7" event={"ID":"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9","Type":"ContainerStarted","Data":"9237d4d881836719668103fdb14453400e68ca740475895d96f68cc4f7b9c43a"}
Oct 30 14:43:37.512971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:37.512849    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:37.562979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:37.562929    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7" podStartSLOduration=2.103639948 podStartE2EDuration="2.562915883s" totalImagesPullingTime="459.275935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:37.562792934 +0000 UTC m=+3282.594691872" watchObservedRunningTime="2026-10-30 14:43:37.562915883 +0000 UTC m=+3282.594814822"
Oct 30 14:43:40.278364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.278330    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:43:40.322764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.322728    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-certs\") pod \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") "
Oct 30 14:43:40.322934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.322783    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-credential-socket\") pod \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") "
Oct 30 14:43:40.322934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.322854    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-socket\") pod \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") "
Oct 30 14:43:40.322934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.322888    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-podinfo\") pod \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") "
Oct 30 14:43:40.322934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.322913    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istiod-ca-cert\") pod \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") "
Oct 30 14:43:40.323145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.322937    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-envoy\") pod \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") "
Oct 30 14:43:40.323145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.323034    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-certs" pod "6bc59ec5-9f0f-481b-8d34-f5e22619f51e" (UID: "6bc59ec5-9f0f-481b-8d34-f5e22619f51e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:40.323272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.323157    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-data\") pod \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") "
Oct 30 14:43:40.323272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.323148    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-credential-socket" pod "6bc59ec5-9f0f-481b-8d34-f5e22619f51e" (UID: "6bc59ec5-9f0f-481b-8d34-f5e22619f51e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:40.323272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.323195    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-token\" (UniqueName: \"kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-token\") pod \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") "
Oct 30 14:43:40.323272 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.323234    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-socket" pod "6bc59ec5-9f0f-481b-8d34-f5e22619f51e" (UID: "6bc59ec5-9f0f-481b-8d34-f5e22619f51e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:40.323491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.323284    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-kube-api-access-8w4h9\" (UniqueName: \"kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-kube-api-access-8w4h9\") pod \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\" (UID: \"6bc59ec5-9f0f-481b-8d34-f5e22619f51e\") "
Oct 30 14:43:40.323491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.323416    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-data" pod "6bc59ec5-9f0f-481b-8d34-f5e22619f51e" (UID: "6bc59ec5-9f0f-481b-8d34-f5e22619f51e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:40.323491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.323323    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istiod-ca-cert" pod "6bc59ec5-9f0f-481b-8d34-f5e22619f51e" (UID: "6bc59ec5-9f0f-481b-8d34-f5e22619f51e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:43:40.323961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.323938    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:40.324119 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.324098    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:40.324119 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.324116    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:40.324306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.324129    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:40.324306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.324141    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:40.325363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.325335    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-token" pod "6bc59ec5-9f0f-481b-8d34-f5e22619f51e" (UID: "6bc59ec5-9f0f-481b-8d34-f5e22619f51e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:43:40.325528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.325507    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-kube-api-access-8w4h9" pod "6bc59ec5-9f0f-481b-8d34-f5e22619f51e" (UID: "6bc59ec5-9f0f-481b-8d34-f5e22619f51e"). InnerVolumeSpecName "kube-api-access-8w4h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:43:40.325721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.325700    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-podinfo" pod "6bc59ec5-9f0f-481b-8d34-f5e22619f51e" (UID: "6bc59ec5-9f0f-481b-8d34-f5e22619f51e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:43:40.325772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.325729    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-envoy" pod "6bc59ec5-9f0f-481b-8d34-f5e22619f51e" (UID: "6bc59ec5-9f0f-481b-8d34-f5e22619f51e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:43:40.425316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.425278    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:40.425316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.425311    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w4h9\" (UniqueName: \"kubernetes.io/projected/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-kube-api-access-8w4h9\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:40.425316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.425322    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:40.425581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.425330    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc59ec5-9f0f-481b-8d34-f5e22619f51e-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:43:40.540595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.540456    4530 generic.go:356] "Generic (PLEG): container finished" podID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerID="036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea" exitCode=0
Oct 30 14:43:40.540595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.540517    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"
Oct 30 14:43:40.540595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.540558    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" event={"ID":"6bc59ec5-9f0f-481b-8d34-f5e22619f51e","Type":"ContainerDied","Data":"036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea"}
Oct 30 14:43:40.540920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.540623    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4" event={"ID":"6bc59ec5-9f0f-481b-8d34-f5e22619f51e","Type":"ContainerDied","Data":"4d9f0ee031c16eb492155aacb481ac4f85b8ad4d02fcd2f39ad77992737731f3"}
Oct 30 14:43:40.540920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.540657    4530 scope.go:122] "RemoveContainer" containerID="036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea"
Oct 30 14:43:40.553897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.553880    4530 scope.go:122] "RemoveContainer" containerID="16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516"
Oct 30 14:43:40.564370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.564353    4530 scope.go:122] "RemoveContainer" containerID="036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea"
Oct 30 14:43:40.564720 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:43:40.564697    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea\": container with ID starting with 036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea not found: ID does not exist" containerID="036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea"
Oct 30 14:43:40.564778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.564729    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea"} err="failed to get container status \"036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea\": rpc error: code = NotFound desc = could not find container \"036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea\": container with ID starting with 036f3f3e66c8b06c714bae3a9c203895798b0652783e53224a536028d568c0ea not found: ID does not exist"
Oct 30 14:43:40.564778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.564749    4530 scope.go:122] "RemoveContainer" containerID="16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516"
Oct 30 14:43:40.565016 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:43:40.564995    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516\": container with ID starting with 16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516 not found: ID does not exist" containerID="16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516"
Oct 30 14:43:40.565070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.565022    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516"} err="failed to get container status \"16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516\": rpc error: code = NotFound desc = could not find container \"16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516\": container with ID starting with 16ebb96ded15678fff036894dde7f62a53687e28f6f1fa68d903b4407dda2516 not found: ID does not exist"
Oct 30 14:43:40.608369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.608346    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"]
Oct 30 14:43:40.613061 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:40.613039    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-4p5h4"]
Oct 30 14:43:41.116433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:41.116395    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" path="/var/lib/kubelet/pods/6bc59ec5-9f0f-481b-8d34-f5e22619f51e/volumes"
Oct 30 14:43:41.368200 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:43:41.368109    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:43:43.528506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:43.528475    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:43:56.088529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:56.088493    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:43:56.090800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:56.090775    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:43:56.093939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:56.093918    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:43:56.094897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:56.094877    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:43:56.095880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:56.095859    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:43:56.096797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:43:56.096781    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:44:04.450605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:04.450556    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"]
Oct 30 14:44:04.451098 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:04.450780    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s" podUID="dc263df1-ccb8-44d7-9f21-a065c9aa64c0" containerName="istio-proxy" containerID="cri-o://df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423" gracePeriod=30
Oct 30 14:44:09.750104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.750077    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:44:09.824249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.824212    4530 generic.go:356] "Generic (PLEG): container finished" podID="dc263df1-ccb8-44d7-9f21-a065c9aa64c0" containerID="df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423" exitCode=0
Oct 30 14:44:09.824466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.824276    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"
Oct 30 14:44:09.824466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.824294    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s" event={"ID":"dc263df1-ccb8-44d7-9f21-a065c9aa64c0","Type":"ContainerDied","Data":"df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423"}
Oct 30 14:44:09.824466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.824374    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s" event={"ID":"dc263df1-ccb8-44d7-9f21-a065c9aa64c0","Type":"ContainerDied","Data":"a9820ecc6c2ed39f22341f9b6351d297e3e748adf91d64b4fed317d2ed552c99"}
Oct 30 14:44:09.824466 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.824407    4530 scope.go:122] "RemoveContainer" containerID="df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423"
Oct 30 14:44:09.841719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.841704    4530 scope.go:122] "RemoveContainer" containerID="df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423"
Oct 30 14:44:09.842073 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:44:09.842044    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423\": container with ID starting with df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423 not found: ID does not exist" containerID="df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423"
Oct 30 14:44:09.842209 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.842086    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423"} err="failed to get container status \"df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423\": rpc error: code = NotFound desc = could not find container \"df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423\": container with ID starting with df9ee6aa473b3c4a8bf35989eff53dd4dc44eba1a106043c95ca2c66181e2423 not found: ID does not exist"
Oct 30 14:44:09.883859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.883830    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-token\" (UniqueName: \"kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-token\") pod \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") "
Oct 30 14:44:09.884025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.883931    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-kube-api-access-lmm25\" (UniqueName: \"kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-kube-api-access-lmm25\") pod \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") "
Oct 30 14:44:09.884025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.883964    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-envoy\") pod \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") "
Oct 30 14:44:09.884147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.884022    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-podinfo\") pod \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") "
Oct 30 14:44:09.884205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.884176    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-socket\") pod \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") "
Oct 30 14:44:09.884267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.884253    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-credential-socket\") pod \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") "
Oct 30 14:44:09.884337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.884287    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-certs\") pod \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") "
Oct 30 14:44:09.884430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.884416    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istiod-ca-cert\") pod \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") "
Oct 30 14:44:09.884554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.884499    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-data\") pod \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\" (UID: \"dc263df1-ccb8-44d7-9f21-a065c9aa64c0\") "
Oct 30 14:44:09.884554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.884523    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-socket" pod "dc263df1-ccb8-44d7-9f21-a065c9aa64c0" (UID: "dc263df1-ccb8-44d7-9f21-a065c9aa64c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:44:09.884674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.884554    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-credential-socket" pod "dc263df1-ccb8-44d7-9f21-a065c9aa64c0" (UID: "dc263df1-ccb8-44d7-9f21-a065c9aa64c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:44:09.884674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.884655    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-certs" pod "dc263df1-ccb8-44d7-9f21-a065c9aa64c0" (UID: "dc263df1-ccb8-44d7-9f21-a065c9aa64c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:44:09.884971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.884954    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-data" pod "dc263df1-ccb8-44d7-9f21-a065c9aa64c0" (UID: "dc263df1-ccb8-44d7-9f21-a065c9aa64c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:44:09.885039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.884993    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istiod-ca-cert" pod "dc263df1-ccb8-44d7-9f21-a065c9aa64c0" (UID: "dc263df1-ccb8-44d7-9f21-a065c9aa64c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:44:09.885197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.885169    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:09.885269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.885197    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:09.885269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.885212    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:09.885269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.885225    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:09.885269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.885237    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:09.886414 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.886386    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-token" pod "dc263df1-ccb8-44d7-9f21-a065c9aa64c0" (UID: "dc263df1-ccb8-44d7-9f21-a065c9aa64c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:44:09.886716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.886697    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-podinfo" pod "dc263df1-ccb8-44d7-9f21-a065c9aa64c0" (UID: "dc263df1-ccb8-44d7-9f21-a065c9aa64c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:44:09.886796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.886722    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-envoy" pod "dc263df1-ccb8-44d7-9f21-a065c9aa64c0" (UID: "dc263df1-ccb8-44d7-9f21-a065c9aa64c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:44:09.886796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.886759    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-kube-api-access-lmm25" pod "dc263df1-ccb8-44d7-9f21-a065c9aa64c0" (UID: "dc263df1-ccb8-44d7-9f21-a065c9aa64c0"). InnerVolumeSpecName "kube-api-access-lmm25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:44:09.986264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.986181    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:09.986264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.986211    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmm25\" (UniqueName: \"kubernetes.io/projected/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-kube-api-access-lmm25\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:09.986264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.986221    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:09.986264 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:09.986230    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc263df1-ccb8-44d7-9f21-a065c9aa64c0-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:10.181653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:10.181621    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"]
Oct 30 14:44:10.191508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:10.191476    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-p4t2s"]
Oct 30 14:44:11.148348 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:11.146863    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc263df1-ccb8-44d7-9f21-a065c9aa64c0" path="/var/lib/kubelet/pods/dc263df1-ccb8-44d7-9f21-a065c9aa64c0/volumes"
Oct 30 14:44:11.153996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:11.153961    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:44:11.368637 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:44:11.368596    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:44:13.376667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.376634    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"]
Oct 30 14:44:13.377512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.377483    4530 cpu_manager.go:414] "containerMap: removing container" podUID="dc263df1-ccb8-44d7-9f21-a065c9aa64c0" containerName="istio-proxy"
Oct 30 14:44:13.377512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.377512    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc263df1-ccb8-44d7-9f21-a065c9aa64c0" containerName="istio-proxy"
Oct 30 14:44:13.377748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.377569    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy"
Oct 30 14:44:13.377748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.377575    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy"
Oct 30 14:44:13.377748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.377589    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy"
Oct 30 14:44:13.377748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.377596    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy"
Oct 30 14:44:13.377748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.377618    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy"
Oct 30 14:44:13.377748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.377623    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy"
Oct 30 14:44:13.378063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.377757    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2abb44c-4a6d-44b2-8081-d24b067dda00" containerName="istio-proxy"
Oct 30 14:44:13.378063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.377765    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy"
Oct 30 14:44:13.378063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.377776    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc59ec5-9f0f-481b-8d34-f5e22619f51e" containerName="istio-proxy"
Oct 30 14:44:13.378063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.377786    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc263df1-ccb8-44d7-9f21-a065c9aa64c0" containerName="istio-proxy"
Oct 30 14:44:13.383412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.383380    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.406830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.406794    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"]
Oct 30 14:44:13.412442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.412413    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-bcc710-openshift-default-dockercfg-6bknp\""
Oct 30 14:44:13.583638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.560284    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-socket\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.583638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.560353    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-podinfo\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.583638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.560403    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-credential-socket\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.583638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.560437    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-data\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.583638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.560473    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlt7k\" (UniqueName: \"kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-kube-api-access-tlt7k\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.583638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.560579    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-token\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.583638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.560699    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-envoy\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.583638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.560741    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-certs\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.583638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.560830    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istiod-ca-cert\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.665631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.665526    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-socket\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.666362 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.666335    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-podinfo\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.690563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.674934    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-credential-socket\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.690563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.674987    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-data\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.690563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.675029    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlt7k\" (UniqueName: \"kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-kube-api-access-tlt7k\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.690563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.675105    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-token\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.690563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.675146    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-envoy\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.690563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.675193    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-certs\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.690563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.675334    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istiod-ca-cert\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.690563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.676251    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istiod-ca-cert\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.690563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.666244    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-socket\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.690563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.676565    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-credential-socket\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.690563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.676823    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-data\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.690563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.678441    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-certs\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.696177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.696148    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-envoy\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.709659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.709625    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-podinfo\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.716330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.716298    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-token\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:13.724180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:13.724145    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlt7k\" (UniqueName: \"kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-kube-api-access-tlt7k\") pod \"e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") " pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:14.013569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:14.013443    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:14.343534 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:44:14.343504    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cb57b5_00f4_4ef0_ad95_c9e99a20f32f.slice/crio-50c03f4f56fc2435aa86da42618d2e7106ecc0c78072175e71d8751bb5ed9a05 WatchSource:0}: Error finding container 50c03f4f56fc2435aa86da42618d2e7106ecc0c78072175e71d8751bb5ed9a05: Status 404 returned error can't find the container with id 50c03f4f56fc2435aa86da42618d2e7106ecc0c78072175e71d8751bb5ed9a05
Oct 30 14:44:14.344571 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:14.344510    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"]
Oct 30 14:44:14.347047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:14.347009    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:44:14.347151 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:14.347078    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:44:14.347151 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:14.347107    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:44:14.871990 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:14.871948    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" event={"ID":"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f","Type":"ContainerStarted","Data":"20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1"}
Oct 30 14:44:14.872436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:14.872049    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" event={"ID":"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f","Type":"ContainerStarted","Data":"50c03f4f56fc2435aa86da42618d2e7106ecc0c78072175e71d8751bb5ed9a05"}
Oct 30 14:44:14.939641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:14.939585    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" podStartSLOduration=1.939570622 podStartE2EDuration="1.939570622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:44:14.937043861 +0000 UTC m=+3319.968942801" watchObservedRunningTime="2026-10-30 14:44:14.939570622 +0000 UTC m=+3319.971469560"
Oct 30 14:44:15.014009 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:15.013973    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:15.020601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:15.020576    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:15.880333 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:15.880300    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:15.881336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:15.881312    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:44:21.595299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.595263    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"]
Oct 30 14:44:21.606486 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.606352    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"]
Oct 30 14:44:21.606865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.606843    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.659824 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.659791    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.660017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.659956    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk8gg\" (UniqueName: \"kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-kube-api-access-nk8gg\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.660017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.659994    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.660099 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.660014    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.660149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.660119    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.660190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.660156    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.660237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.660184    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.660287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.660237    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.660287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.660260    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.706026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.705992    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d05e21-openshift-default-dockercfg-6p687\""
Oct 30 14:44:21.783565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.782580    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.819567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.817748    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk8gg\" (UniqueName: \"kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-kube-api-access-nk8gg\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.819567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.818024    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.819567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.818078    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.819567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.818146    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.819567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.818244    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.819567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.818331    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.819567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.818367    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.819567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.818405    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.819567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.818528    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.819567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.818965    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.819567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.819052    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.819567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.819334    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.824561 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.822468    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.866609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.851989    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.866609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.853984    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.866609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.858473    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.866609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.859244    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk8gg\" (UniqueName: \"kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-kube-api-access-nk8gg\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.934574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.934172    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:21.967060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.958551    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"]
Oct 30 14:44:21.981996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.977751    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"
Oct 30 14:44:21.981996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.980831    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"]
Oct 30 14:44:22.000814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.999346    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3b95dd\""
Oct 30 14:44:22.000814 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:21.999417    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3b95dd-dockercfg-kbrdd\""
Oct 30 14:44:22.026234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.026168    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgzvm\" (UniqueName: \"kubernetes.io/projected/c1ab0398-256e-4d59-814a-bde23499d65d-kube-api-access-kgzvm\") pod \"payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s\" (UID: \"c1ab0398-256e-4d59-814a-bde23499d65d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"
Oct 30 14:44:22.026594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.026566    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ab0398-256e-4d59-814a-bde23499d65d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s\" (UID: \"c1ab0398-256e-4d59-814a-bde23499d65d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"
Oct 30 14:44:22.128797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.128703    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzvm\" (UniqueName: \"kubernetes.io/projected/c1ab0398-256e-4d59-814a-bde23499d65d-kube-api-access-kgzvm\") pod \"payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s\" (UID: \"c1ab0398-256e-4d59-814a-bde23499d65d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"
Oct 30 14:44:22.128797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.128773    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ab0398-256e-4d59-814a-bde23499d65d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s\" (UID: \"c1ab0398-256e-4d59-814a-bde23499d65d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"
Oct 30 14:44:22.130040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.130010    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ab0398-256e-4d59-814a-bde23499d65d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s\" (UID: \"c1ab0398-256e-4d59-814a-bde23499d65d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"
Oct 30 14:44:22.156796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.156767    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzvm\" (UniqueName: \"kubernetes.io/projected/c1ab0398-256e-4d59-814a-bde23499d65d-kube-api-access-kgzvm\") pod \"payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s\" (UID: \"c1ab0398-256e-4d59-814a-bde23499d65d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"
Oct 30 14:44:22.318700 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:44:22.318667    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a82d8a8_0574_4500_80f1_6fdd38b4c8b1.slice/crio-11dea62eaddbbb0c7b518ca20fe9f320b0a251feee6a5b042236b09a27ef2a90 WatchSource:0}: Error finding container 11dea62eaddbbb0c7b518ca20fe9f320b0a251feee6a5b042236b09a27ef2a90: Status 404 returned error can't find the container with id 11dea62eaddbbb0c7b518ca20fe9f320b0a251feee6a5b042236b09a27ef2a90
Oct 30 14:44:22.319933 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.319907    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"]
Oct 30 14:44:22.321738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.321701    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:44:22.321816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.321772    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:44:22.321816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.321800    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:44:22.324295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.324271    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"
Oct 30 14:44:22.653849 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:44:22.653813    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ab0398_256e_4d59_814a_bde23499d65d.slice/crio-8c313ec82b9155d64ecb24d22060cd9878a2d1d2b0729d5693f38e760b3fe203 WatchSource:0}: Error finding container 8c313ec82b9155d64ecb24d22060cd9878a2d1d2b0729d5693f38e760b3fe203: Status 404 returned error can't find the container with id 8c313ec82b9155d64ecb24d22060cd9878a2d1d2b0729d5693f38e760b3fe203
Oct 30 14:44:22.671371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.671313    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"]
Oct 30 14:44:22.959022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.958922    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv" event={"ID":"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1","Type":"ContainerStarted","Data":"06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d"}
Oct 30 14:44:22.959022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.958985    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv" event={"ID":"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1","Type":"ContainerStarted","Data":"11dea62eaddbbb0c7b518ca20fe9f320b0a251feee6a5b042236b09a27ef2a90"}
Oct 30 14:44:22.960791 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.960755    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s" event={"ID":"c1ab0398-256e-4d59-814a-bde23499d65d","Type":"ContainerStarted","Data":"0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10"}
Oct 30 14:44:22.960934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.960809    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s" event={"ID":"c1ab0398-256e-4d59-814a-bde23499d65d","Type":"ContainerStarted","Data":"8c313ec82b9155d64ecb24d22060cd9878a2d1d2b0729d5693f38e760b3fe203"}
Oct 30 14:44:22.960934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:22.960847    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"
Oct 30 14:44:23.015361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:23.015305    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv" podStartSLOduration=2.015290847 podStartE2EDuration="2.015290847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:44:23.010318478 +0000 UTC m=+3328.042217431" watchObservedRunningTime="2026-10-30 14:44:23.015290847 +0000 UTC m=+3328.047189861"
Oct 30 14:44:23.061140 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:23.061092    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s" podStartSLOduration=2.061077968 podStartE2EDuration="2.061077968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:44:23.057089108 +0000 UTC m=+3328.088988051" watchObservedRunningTime="2026-10-30 14:44:23.061077968 +0000 UTC m=+3328.092976906"
Oct 30 14:44:23.935839 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:23.935797    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:23.937732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:23.937699    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 14:44:23.937885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:23.937754    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv" podUID="9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 14:44:24.937978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:24.937935    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 14:44:24.938671 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:24.938636    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv" podUID="9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 14:44:25.935275 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:25.935230    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 14:44:25.935551 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:25.935298    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv" podUID="9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 14:44:26.936193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:26.936155    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 14:44:26.936633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:26.936215    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv" podUID="9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 14:44:27.935654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:27.935617    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 14:44:27.935860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:27.935678    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv" podUID="9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 14:44:28.935203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:28.935165    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 14:44:28.935835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:28.935228    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv" podUID="9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 14:44:29.941443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:29.941409    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:29.942040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:29.942016    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:29.942895 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:29.942875    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:44:33.972130 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:33.972096    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"
Oct 30 14:44:41.369063 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:44:41.369023    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:44:45.424854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:45.424819    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"]
Oct 30 14:44:45.425369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:45.425075    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb" podUID="e1f64945-8288-4cff-8760-118acf8e4427" containerName="maas-api" containerID="cri-o://faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26" gracePeriod=30
Oct 30 14:44:45.826452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:45.826407    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:44:46.057171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.047291    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-maas-api-tls\") pod \"e1f64945-8288-4cff-8760-118acf8e4427\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") "
Oct 30 14:44:46.057171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.047413    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-metrics-certs\") pod \"e1f64945-8288-4cff-8760-118acf8e4427\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") "
Oct 30 14:44:46.057171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.047475    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f64945-8288-4cff-8760-118acf8e4427-kube-api-access-c4zqm\" (UniqueName: \"kubernetes.io/projected/e1f64945-8288-4cff-8760-118acf8e4427-kube-api-access-c4zqm\") pod \"e1f64945-8288-4cff-8760-118acf8e4427\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") "
Oct 30 14:44:46.080922 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.080882    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f64945-8288-4cff-8760-118acf8e4427-kube-api-access-c4zqm" pod "e1f64945-8288-4cff-8760-118acf8e4427" (UID: "e1f64945-8288-4cff-8760-118acf8e4427"). InnerVolumeSpecName "kube-api-access-c4zqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:44:46.082815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.082768    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-maas-api-tls" pod "e1f64945-8288-4cff-8760-118acf8e4427" (UID: "e1f64945-8288-4cff-8760-118acf8e4427"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:44:46.087330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.084072    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f64945-8288-4cff-8760-118acf8e4427-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f64945-8288-4cff-8760-118acf8e4427-openshift-service-ca\") pod \"e1f64945-8288-4cff-8760-118acf8e4427\" (UID: \"e1f64945-8288-4cff-8760-118acf8e4427\") "
Oct 30 14:44:46.087330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.084880    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:46.087330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.084905    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4zqm\" (UniqueName: \"kubernetes.io/projected/e1f64945-8288-4cff-8760-118acf8e4427-kube-api-access-c4zqm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:46.087330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.085378    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f64945-8288-4cff-8760-118acf8e4427-openshift-service-ca" pod "e1f64945-8288-4cff-8760-118acf8e4427" (UID: "e1f64945-8288-4cff-8760-118acf8e4427"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:44:46.104753 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.104700    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-metrics-certs" pod "e1f64945-8288-4cff-8760-118acf8e4427" (UID: "e1f64945-8288-4cff-8760-118acf8e4427"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:44:46.186695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.186661    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f64945-8288-4cff-8760-118acf8e4427-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:46.186695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.186690    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f64945-8288-4cff-8760-118acf8e4427-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:44:46.351441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.351345    4530 generic.go:356] "Generic (PLEG): container finished" podID="e1f64945-8288-4cff-8760-118acf8e4427" containerID="faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26" exitCode=0
Oct 30 14:44:46.351441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.351397    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"
Oct 30 14:44:46.351441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.351427    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb" event={"ID":"e1f64945-8288-4cff-8760-118acf8e4427","Type":"ContainerDied","Data":"faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26"}
Oct 30 14:44:46.351730 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.351483    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb" event={"ID":"e1f64945-8288-4cff-8760-118acf8e4427","Type":"ContainerDied","Data":"f97b2578ac1fa0404d34593de51a8683d695f17142909af009db954805488464"}
Oct 30 14:44:46.351730 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.351505    4530 scope.go:122] "RemoveContainer" containerID="faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26"
Oct 30 14:44:46.363253 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.363236    4530 scope.go:122] "RemoveContainer" containerID="faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26"
Oct 30 14:44:46.363531 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:44:46.363510    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26\": container with ID starting with faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26 not found: ID does not exist" containerID="faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26"
Oct 30 14:44:46.363607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.363561    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26"} err="failed to get container status \"faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26\": rpc error: code = NotFound desc = could not find container \"faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26\": container with ID starting with faccf839b1591a2bfcc1d1efbd7e3031033bb8fa9720ad8d4615f2a45b255d26 not found: ID does not exist"
Oct 30 14:44:46.437837 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.437662    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"]
Oct 30 14:44:46.468268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:46.468224    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e50e63-858457b9ff-f5hbb"]
Oct 30 14:44:47.116879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:47.116839    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f64945-8288-4cff-8760-118acf8e4427" path="/var/lib/kubelet/pods/e1f64945-8288-4cff-8760-118acf8e4427/volumes"
Oct 30 14:44:54.785241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:54.785199    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"]
Oct 30 14:44:54.785761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:54.785498    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" podUID="43206177-2c02-4a0f-87da-a00340846626" containerName="istio-proxy" containerID="cri-o://390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd" gracePeriod=30
Oct 30 14:44:58.898936 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:58.898895    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"]
Oct 30 14:44:58.899444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:58.899181    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy" containerID="cri-o://36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685" gracePeriod=30
Oct 30 14:44:59.722516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:44:59.722043    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:45:00.082066 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.082037    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:45:00.190589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.189635    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-socket\") pod \"43206177-2c02-4a0f-87da-a00340846626\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") "
Oct 30 14:45:00.190589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.189702    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-istio-token\" (UniqueName: \"kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-istio-token\") pod \"43206177-2c02-4a0f-87da-a00340846626\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") "
Oct 30 14:45:00.190589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.190050    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-socket" pod "43206177-2c02-4a0f-87da-a00340846626" (UID: "43206177-2c02-4a0f-87da-a00340846626"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:00.190589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.190168    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-credential-socket\") pod \"43206177-2c02-4a0f-87da-a00340846626\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") "
Oct 30 14:45:00.190589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.190229    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-envoy\") pod \"43206177-2c02-4a0f-87da-a00340846626\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") "
Oct 30 14:45:00.190589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.190310    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-certs\") pod \"43206177-2c02-4a0f-87da-a00340846626\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") "
Oct 30 14:45:00.190589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.190342    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-kube-api-access-dm8dw\" (UniqueName: \"kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-kube-api-access-dm8dw\") pod \"43206177-2c02-4a0f-87da-a00340846626\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") "
Oct 30 14:45:00.190589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.190405    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43206177-2c02-4a0f-87da-a00340846626-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43206177-2c02-4a0f-87da-a00340846626-istiod-ca-cert\") pod \"43206177-2c02-4a0f-87da-a00340846626\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") "
Oct 30 14:45:00.190589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.190462    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43206177-2c02-4a0f-87da-a00340846626-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43206177-2c02-4a0f-87da-a00340846626-istio-podinfo\") pod \"43206177-2c02-4a0f-87da-a00340846626\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") "
Oct 30 14:45:00.190589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.190495    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-credential-socket" pod "43206177-2c02-4a0f-87da-a00340846626" (UID: "43206177-2c02-4a0f-87da-a00340846626"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:00.190589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.190564    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-data\") pod \"43206177-2c02-4a0f-87da-a00340846626\" (UID: \"43206177-2c02-4a0f-87da-a00340846626\") "
Oct 30 14:45:00.191372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.190726    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-certs" pod "43206177-2c02-4a0f-87da-a00340846626" (UID: "43206177-2c02-4a0f-87da-a00340846626"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:00.191372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.190966    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-data" pod "43206177-2c02-4a0f-87da-a00340846626" (UID: "43206177-2c02-4a0f-87da-a00340846626"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:00.191372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.191335    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:00.191372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.191356    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:00.191372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.191370    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:00.191718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.191387    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:00.192244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.192208    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43206177-2c02-4a0f-87da-a00340846626-istiod-ca-cert" pod "43206177-2c02-4a0f-87da-a00340846626" (UID: "43206177-2c02-4a0f-87da-a00340846626"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:45:00.196699 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.196661    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-kube-api-access-dm8dw" pod "43206177-2c02-4a0f-87da-a00340846626" (UID: "43206177-2c02-4a0f-87da-a00340846626"). InnerVolumeSpecName "kube-api-access-dm8dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:00.198408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.198379    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-istio-token" pod "43206177-2c02-4a0f-87da-a00340846626" (UID: "43206177-2c02-4a0f-87da-a00340846626"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:00.198951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.198892    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43206177-2c02-4a0f-87da-a00340846626-istio-podinfo" pod "43206177-2c02-4a0f-87da-a00340846626" (UID: "43206177-2c02-4a0f-87da-a00340846626"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:45:00.201971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.201945    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-envoy" pod "43206177-2c02-4a0f-87da-a00340846626" (UID: "43206177-2c02-4a0f-87da-a00340846626"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:00.210448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.210412    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"]
Oct 30 14:45:00.213263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.213238    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e1f64945-8288-4cff-8760-118acf8e4427" containerName="maas-api"
Oct 30 14:45:00.213409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.213390    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f64945-8288-4cff-8760-118acf8e4427" containerName="maas-api"
Oct 30 14:45:00.213596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.213581    4530 cpu_manager.go:414] "containerMap: removing container" podUID="43206177-2c02-4a0f-87da-a00340846626" containerName="istio-proxy"
Oct 30 14:45:00.213707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.213694    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43206177-2c02-4a0f-87da-a00340846626" containerName="istio-proxy"
Oct 30 14:45:00.213816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.213804    4530 cpu_manager.go:414] "containerMap: removing container" podUID="43206177-2c02-4a0f-87da-a00340846626" containerName="istio-proxy"
Oct 30 14:45:00.213910 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.213898    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43206177-2c02-4a0f-87da-a00340846626" containerName="istio-proxy"
Oct 30 14:45:00.214245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.214222    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="43206177-2c02-4a0f-87da-a00340846626" containerName="istio-proxy"
Oct 30 14:45:00.214356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.214248    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="43206177-2c02-4a0f-87da-a00340846626" containerName="istio-proxy"
Oct 30 14:45:00.214356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.214273    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f64945-8288-4cff-8760-118acf8e4427" containerName="maas-api"
Oct 30 14:45:00.220039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.219249    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"
Oct 30 14:45:00.245315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.244847    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"]
Oct 30 14:45:00.253226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.253197    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"]
Oct 30 14:45:00.258965 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.258942    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"
Oct 30 14:45:00.284122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.284084    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"]
Oct 30 14:45:00.291031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.290966    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"
Oct 30 14:45:00.295659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.292385    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0686b700-1a3d-4a49-82e0-73d5ac9de450-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2\" (UID: \"0686b700-1a3d-4a49-82e0-73d5ac9de450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"
Oct 30 14:45:00.295659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.292532    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wmf\" (UniqueName: \"kubernetes.io/projected/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-kube-api-access-67wmf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5\" (UID: \"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"
Oct 30 14:45:00.295659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.292666    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4hmm\" (UniqueName: \"kubernetes.io/projected/6bd36ed1-3a87-445f-a966-77792741b82c-kube-api-access-w4hmm\") pod \"maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq\" (UID: \"6bd36ed1-3a87-445f-a966-77792741b82c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"
Oct 30 14:45:00.295659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.292700    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmk5n\" (UniqueName: \"kubernetes.io/projected/0686b700-1a3d-4a49-82e0-73d5ac9de450-kube-api-access-kmk5n\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2\" (UID: \"0686b700-1a3d-4a49-82e0-73d5ac9de450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"
Oct 30 14:45:00.295659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.292800    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd36ed1-3a87-445f-a966-77792741b82c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq\" (UID: \"6bd36ed1-3a87-445f-a966-77792741b82c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"
Oct 30 14:45:00.295659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.292912    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5\" (UID: \"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"
Oct 30 14:45:00.295659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.293038    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43206177-2c02-4a0f-87da-a00340846626-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:00.295659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.293055    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43206177-2c02-4a0f-87da-a00340846626-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:00.295659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.293067    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:00.295659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.293081    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43206177-2c02-4a0f-87da-a00340846626-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:00.295659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.293094    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm8dw\" (UniqueName: \"kubernetes.io/projected/43206177-2c02-4a0f-87da-a00340846626-kube-api-access-dm8dw\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:00.299641 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.299486    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"]
Oct 30 14:45:00.308092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.307611    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:45:00.311917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.308323    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"]
Oct 30 14:45:00.315944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.314422    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"
Oct 30 14:45:00.316958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.316932    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"]
Oct 30 14:45:00.323872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.323812    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"]
Oct 30 14:45:00.329766 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.329552    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"]
Oct 30 14:45:00.337707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.337638    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"
Oct 30 14:45:00.338867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.338847    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"]
Oct 30 14:45:00.348840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.345689    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"]
Oct 30 14:45:00.350729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.350710    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"
Oct 30 14:45:00.353826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.353806    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"]
Oct 30 14:45:00.380577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.378797    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"]
Oct 30 14:45:00.394311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.394281    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0686b700-1a3d-4a49-82e0-73d5ac9de450-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2\" (UID: \"0686b700-1a3d-4a49-82e0-73d5ac9de450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"
Oct 30 14:45:00.394464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.394350    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjf9k\" (UniqueName: \"kubernetes.io/projected/24867fc7-ab36-491e-bccf-64a34aa01681-kube-api-access-tjf9k\") pod \"maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf\" (UID: \"24867fc7-ab36-491e-bccf-64a34aa01681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"
Oct 30 14:45:00.394464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.394374    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9827\" (UniqueName: \"kubernetes.io/projected/b88f1bbb-88c1-4fff-9987-0e9ed0699865-kube-api-access-j9827\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp\" (UID: \"b88f1bbb-88c1-4fff-9987-0e9ed0699865\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"
Oct 30 14:45:00.394464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.394395    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wmf\" (UniqueName: \"kubernetes.io/projected/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-kube-api-access-67wmf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5\" (UID: \"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"
Oct 30 14:45:00.394464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.394423    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24867fc7-ab36-491e-bccf-64a34aa01681-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf\" (UID: \"24867fc7-ab36-491e-bccf-64a34aa01681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"
Oct 30 14:45:00.394464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.394451    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4hmm\" (UniqueName: \"kubernetes.io/projected/6bd36ed1-3a87-445f-a966-77792741b82c-kube-api-access-w4hmm\") pod \"maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq\" (UID: \"6bd36ed1-3a87-445f-a966-77792741b82c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"
Oct 30 14:45:00.394773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.394469    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmk5n\" (UniqueName: \"kubernetes.io/projected/0686b700-1a3d-4a49-82e0-73d5ac9de450-kube-api-access-kmk5n\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2\" (UID: \"0686b700-1a3d-4a49-82e0-73d5ac9de450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"
Oct 30 14:45:00.394773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.394493    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a608b93-87cb-4d9f-9d5a-8414d844b98e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6\" (UID: \"7a608b93-87cb-4d9f-9d5a-8414d844b98e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"
Oct 30 14:45:00.394773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.394511    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88f1bbb-88c1-4fff-9987-0e9ed0699865-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp\" (UID: \"b88f1bbb-88c1-4fff-9987-0e9ed0699865\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"
Oct 30 14:45:00.394773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.394583    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt4zc\" (UniqueName: \"kubernetes.io/projected/7a608b93-87cb-4d9f-9d5a-8414d844b98e-kube-api-access-vt4zc\") pod \"maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6\" (UID: \"7a608b93-87cb-4d9f-9d5a-8414d844b98e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"
Oct 30 14:45:00.394773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.394609    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd36ed1-3a87-445f-a966-77792741b82c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq\" (UID: \"6bd36ed1-3a87-445f-a966-77792741b82c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"
Oct 30 14:45:00.394773 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.394679    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5\" (UID: \"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"
Oct 30 14:45:00.395413 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.395383    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5\" (UID: \"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"
Oct 30 14:45:00.396867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.396842    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd36ed1-3a87-445f-a966-77792741b82c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq\" (UID: \"6bd36ed1-3a87-445f-a966-77792741b82c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"
Oct 30 14:45:00.397444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.397421    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0686b700-1a3d-4a49-82e0-73d5ac9de450-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2\" (UID: \"0686b700-1a3d-4a49-82e0-73d5ac9de450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"
Oct 30 14:45:00.428052 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.428028    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wmf\" (UniqueName: \"kubernetes.io/projected/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-kube-api-access-67wmf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5\" (UID: \"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"
Oct 30 14:45:00.428211 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.428178    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmk5n\" (UniqueName: \"kubernetes.io/projected/0686b700-1a3d-4a49-82e0-73d5ac9de450-kube-api-access-kmk5n\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2\" (UID: \"0686b700-1a3d-4a49-82e0-73d5ac9de450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"
Oct 30 14:45:00.428595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.428574    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4hmm\" (UniqueName: \"kubernetes.io/projected/6bd36ed1-3a87-445f-a966-77792741b82c-kube-api-access-w4hmm\") pod \"maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq\" (UID: \"6bd36ed1-3a87-445f-a966-77792741b82c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"
Oct 30 14:45:00.508298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.496780    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt4zc\" (UniqueName: \"kubernetes.io/projected/7a608b93-87cb-4d9f-9d5a-8414d844b98e-kube-api-access-vt4zc\") pod \"maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6\" (UID: \"7a608b93-87cb-4d9f-9d5a-8414d844b98e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"
Oct 30 14:45:00.508298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.497055    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjf9k\" (UniqueName: \"kubernetes.io/projected/24867fc7-ab36-491e-bccf-64a34aa01681-kube-api-access-tjf9k\") pod \"maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf\" (UID: \"24867fc7-ab36-491e-bccf-64a34aa01681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"
Oct 30 14:45:00.508298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.497092    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9827\" (UniqueName: \"kubernetes.io/projected/b88f1bbb-88c1-4fff-9987-0e9ed0699865-kube-api-access-j9827\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp\" (UID: \"b88f1bbb-88c1-4fff-9987-0e9ed0699865\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"
Oct 30 14:45:00.508298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.497134    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24867fc7-ab36-491e-bccf-64a34aa01681-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf\" (UID: \"24867fc7-ab36-491e-bccf-64a34aa01681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"
Oct 30 14:45:00.508298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.497190    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a608b93-87cb-4d9f-9d5a-8414d844b98e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6\" (UID: \"7a608b93-87cb-4d9f-9d5a-8414d844b98e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"
Oct 30 14:45:00.508298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.497218    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88f1bbb-88c1-4fff-9987-0e9ed0699865-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp\" (UID: \"b88f1bbb-88c1-4fff-9987-0e9ed0699865\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"
Oct 30 14:45:00.508298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.498504    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88f1bbb-88c1-4fff-9987-0e9ed0699865-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp\" (UID: \"b88f1bbb-88c1-4fff-9987-0e9ed0699865\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"
Oct 30 14:45:00.508298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.498526    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a608b93-87cb-4d9f-9d5a-8414d844b98e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6\" (UID: \"7a608b93-87cb-4d9f-9d5a-8414d844b98e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"
Oct 30 14:45:00.508298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.499036    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24867fc7-ab36-491e-bccf-64a34aa01681-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf\" (UID: \"24867fc7-ab36-491e-bccf-64a34aa01681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"
Oct 30 14:45:00.523398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.523365    4530 generic.go:356] "Generic (PLEG): container finished" podID="43206177-2c02-4a0f-87da-a00340846626" containerID="390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd" exitCode=0
Oct 30 14:45:00.523630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.523421    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"
Oct 30 14:45:00.523630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.523427    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" event={"ID":"43206177-2c02-4a0f-87da-a00340846626","Type":"ContainerDied","Data":"390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd"}
Oct 30 14:45:00.523630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.523485    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn" event={"ID":"43206177-2c02-4a0f-87da-a00340846626","Type":"ContainerDied","Data":"ee00be75870f6d868d35dc10312d5575cc8f9642fcc30c3e12b0ae181e211e1d"}
Oct 30 14:45:00.523630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.523511    4530 scope.go:122] "RemoveContainer" containerID="390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd"
Oct 30 14:45:00.528164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.528139    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9827\" (UniqueName: \"kubernetes.io/projected/b88f1bbb-88c1-4fff-9987-0e9ed0699865-kube-api-access-j9827\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp\" (UID: \"b88f1bbb-88c1-4fff-9987-0e9ed0699865\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"
Oct 30 14:45:00.528300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.528207    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt4zc\" (UniqueName: \"kubernetes.io/projected/7a608b93-87cb-4d9f-9d5a-8414d844b98e-kube-api-access-vt4zc\") pod \"maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6\" (UID: \"7a608b93-87cb-4d9f-9d5a-8414d844b98e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"
Oct 30 14:45:00.528370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.528322    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjf9k\" (UniqueName: \"kubernetes.io/projected/24867fc7-ab36-491e-bccf-64a34aa01681-kube-api-access-tjf9k\") pod \"maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf\" (UID: \"24867fc7-ab36-491e-bccf-64a34aa01681\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"
Oct 30 14:45:00.536401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.536383    4530 scope.go:122] "RemoveContainer" containerID="250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d"
Oct 30 14:45:00.547047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.547021    4530 scope.go:122] "RemoveContainer" containerID="390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd"
Oct 30 14:45:00.547317 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:00.547298    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd\": container with ID starting with 390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd not found: ID does not exist" containerID="390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd"
Oct 30 14:45:00.547365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.547326    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd"} err="failed to get container status \"390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd\": rpc error: code = NotFound desc = could not find container \"390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd\": container with ID starting with 390851f81418a86fddb4d5eb06a44d2152f4824c40edd97fc22fc01c250813bd not found: ID does not exist"
Oct 30 14:45:00.547365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.547346    4530 scope.go:122] "RemoveContainer" containerID="250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d"
Oct 30 14:45:00.547626 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:00.547609    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d\": container with ID starting with 250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d not found: ID does not exist" containerID="250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d"
Oct 30 14:45:00.547702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.547630    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d"} err="failed to get container status \"250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d\": rpc error: code = NotFound desc = could not find container \"250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d\": container with ID starting with 250f2506925555d614d14a168fb288415282773ec396783b0d0c82d43ac4485d not found: ID does not exist"
Oct 30 14:45:00.568991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.568939    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"
Oct 30 14:45:00.600325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.600301    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"
Oct 30 14:45:00.643794 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.643705    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"]
Oct 30 14:45:00.649901 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.649869    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-jdpzn"]
Oct 30 14:45:00.654836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.653265    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"
Oct 30 14:45:00.666786 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.666702    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"
Oct 30 14:45:00.682167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.679501    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"
Oct 30 14:45:00.692715 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.692302    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"
Oct 30 14:45:00.913123 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:45:00.913089    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec96aca_bd5f_4ccc_97e9_d4d63f7bc0b5.slice/crio-6349c9faecd4f8144c1eef4cb63d44f264ed6fe6207b4c103ae431f84628b07d WatchSource:0}: Error finding container 6349c9faecd4f8144c1eef4cb63d44f264ed6fe6207b4c103ae431f84628b07d: Status 404 returned error can't find the container with id 6349c9faecd4f8144c1eef4cb63d44f264ed6fe6207b4c103ae431f84628b07d
Oct 30 14:45:00.915492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:00.915466    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"]
Oct 30 14:45:01.061369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.061342    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"]
Oct 30 14:45:01.073222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.073196    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"]
Oct 30 14:45:01.077525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.077501    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"]
Oct 30 14:45:01.082392 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:45:01.082368    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24867fc7_ab36_491e_bccf_64a34aa01681.slice/crio-bdc2b03c79795b9c94d85bfcd25a382dd5c189b63205242634bcd987315c2e3e WatchSource:0}: Error finding container bdc2b03c79795b9c94d85bfcd25a382dd5c189b63205242634bcd987315c2e3e: Status 404 returned error can't find the container with id bdc2b03c79795b9c94d85bfcd25a382dd5c189b63205242634bcd987315c2e3e
Oct 30 14:45:01.083892 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:45:01.083855    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a608b93_87cb_4d9f_9d5a_8414d844b98e.slice/crio-0588155eaaafd985056e4f7c4ff2479aa5223a585aae36aba20937ab041ecfeb WatchSource:0}: Error finding container 0588155eaaafd985056e4f7c4ff2479aa5223a585aae36aba20937ab041ecfeb: Status 404 returned error can't find the container with id 0588155eaaafd985056e4f7c4ff2479aa5223a585aae36aba20937ab041ecfeb
Oct 30 14:45:01.086088 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:45:01.085925    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd36ed1_3a87_445f_a966_77792741b82c.slice/crio-eac8d04963d55c559858ca03d937f9e53eaf522aea995c5c9b2d5d03086b48a8 WatchSource:0}: Error finding container eac8d04963d55c559858ca03d937f9e53eaf522aea995c5c9b2d5d03086b48a8: Status 404 returned error can't find the container with id eac8d04963d55c559858ca03d937f9e53eaf522aea995c5c9b2d5d03086b48a8
Oct 30 14:45:01.118523 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.118496    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43206177-2c02-4a0f-87da-a00340846626" path="/var/lib/kubelet/pods/43206177-2c02-4a0f-87da-a00340846626/volumes"
Oct 30 14:45:01.153015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.140834    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"]
Oct 30 14:45:01.277681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.277653    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"]
Oct 30 14:45:01.291248 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:45:01.291212    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88f1bbb_88c1_4fff_9987_0e9ed0699865.slice/crio-1e29671f42704b399affe2065173ccb1b39a0009141b084ad9fca489cf3b1d42 WatchSource:0}: Error finding container 1e29671f42704b399affe2065173ccb1b39a0009141b084ad9fca489cf3b1d42: Status 404 returned error can't find the container with id 1e29671f42704b399affe2065173ccb1b39a0009141b084ad9fca489cf3b1d42
Oct 30 14:45:01.636168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.634570    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2" event={"ID":"0686b700-1a3d-4a49-82e0-73d5ac9de450","Type":"ContainerStarted","Data":"534cabb624b180234dc4893c0fc8da985924fa59d485bc345fd4bc8434d6ee0f"}
Oct 30 14:45:01.639490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.639353    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq" event={"ID":"6bd36ed1-3a87-445f-a966-77792741b82c","Type":"ContainerStarted","Data":"eac8d04963d55c559858ca03d937f9e53eaf522aea995c5c9b2d5d03086b48a8"}
Oct 30 14:45:01.644601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.641684    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp" event={"ID":"b88f1bbb-88c1-4fff-9987-0e9ed0699865","Type":"ContainerStarted","Data":"1e29671f42704b399affe2065173ccb1b39a0009141b084ad9fca489cf3b1d42"}
Oct 30 14:45:01.644943 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.644900    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf" event={"ID":"24867fc7-ab36-491e-bccf-64a34aa01681","Type":"ContainerStarted","Data":"bdc2b03c79795b9c94d85bfcd25a382dd5c189b63205242634bcd987315c2e3e"}
Oct 30 14:45:01.653577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.650029    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6" event={"ID":"7a608b93-87cb-4d9f-9d5a-8414d844b98e","Type":"ContainerStarted","Data":"0588155eaaafd985056e4f7c4ff2479aa5223a585aae36aba20937ab041ecfeb"}
Oct 30 14:45:01.654438 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.654416    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5" event={"ID":"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5","Type":"ContainerStarted","Data":"6349c9faecd4f8144c1eef4cb63d44f264ed6fe6207b4c103ae431f84628b07d"}
Oct 30 14:45:01.706603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:01.706505    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5" podStartSLOduration=1.706480028 podStartE2EDuration="1.706480028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:01.69908024 +0000 UTC m=+3366.730979192" watchObservedRunningTime="2026-10-30 14:45:01.706480028 +0000 UTC m=+3366.738379043"
Oct 30 14:45:02.664825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.664785    4530 generic.go:356] "Generic (PLEG): container finished" podID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" containerID="300dbff144db9b9d33075ab2f4690352433e5a5f202c3d024f35a1ac0d342e92" exitCode=60
Oct 30 14:45:02.665330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.664867    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp" event={"ID":"b88f1bbb-88c1-4fff-9987-0e9ed0699865","Type":"ContainerDied","Data":"300dbff144db9b9d33075ab2f4690352433e5a5f202c3d024f35a1ac0d342e92"}
Oct 30 14:45:02.665330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.665205    4530 scope.go:122] "RemoveContainer" containerID="300dbff144db9b9d33075ab2f4690352433e5a5f202c3d024f35a1ac0d342e92"
Oct 30 14:45:02.674032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.674006    4530 generic.go:356] "Generic (PLEG): container finished" podID="24867fc7-ab36-491e-bccf-64a34aa01681" containerID="5b545e54fe48c9aa82a9e4564d361c453c15c15245906bf8e4313d655b886524" exitCode=60
Oct 30 14:45:02.674163 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.674081    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf" event={"ID":"24867fc7-ab36-491e-bccf-64a34aa01681","Type":"ContainerDied","Data":"5b545e54fe48c9aa82a9e4564d361c453c15c15245906bf8e4313d655b886524"}
Oct 30 14:45:02.674311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.674294    4530 scope.go:122] "RemoveContainer" containerID="5b545e54fe48c9aa82a9e4564d361c453c15c15245906bf8e4313d655b886524"
Oct 30 14:45:02.675937 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.675912    4530 generic.go:356] "Generic (PLEG): container finished" podID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" containerID="4f2012cfcf7992e34ac5bbc3383bd9726e498e430212152967d1530f1a2ed1f7" exitCode=60
Oct 30 14:45:02.676028 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.675988    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6" event={"ID":"7a608b93-87cb-4d9f-9d5a-8414d844b98e","Type":"ContainerDied","Data":"4f2012cfcf7992e34ac5bbc3383bd9726e498e430212152967d1530f1a2ed1f7"}
Oct 30 14:45:02.676185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.676166    4530 scope.go:122] "RemoveContainer" containerID="4f2012cfcf7992e34ac5bbc3383bd9726e498e430212152967d1530f1a2ed1f7"
Oct 30 14:45:02.677858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.677838    4530 generic.go:356] "Generic (PLEG): container finished" podID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" containerID="f33c9fbf95062a251c99c37fca40aad85441b7e097976abf97e6f37954b18b3d" exitCode=60
Oct 30 14:45:02.677980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.677942    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5" event={"ID":"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5","Type":"ContainerDied","Data":"f33c9fbf95062a251c99c37fca40aad85441b7e097976abf97e6f37954b18b3d"}
Oct 30 14:45:02.678126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.678109    4530 scope.go:122] "RemoveContainer" containerID="f33c9fbf95062a251c99c37fca40aad85441b7e097976abf97e6f37954b18b3d"
Oct 30 14:45:02.701565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.679765    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2" event={"ID":"0686b700-1a3d-4a49-82e0-73d5ac9de450","Type":"ContainerStarted","Data":"bd9d0a6888ab28a02c7328d408dfbbb8aba73bc96f051b381bfb685b447f9159"}
Oct 30 14:45:02.707554 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.707502    4530 generic.go:356] "Generic (PLEG): container finished" podID="6bd36ed1-3a87-445f-a966-77792741b82c" containerID="3181598ee40e30f4ea2f279adcdd955bd72633dcc84770eb9480bf11be3d1037" exitCode=60
Oct 30 14:45:02.707698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.707676    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq" event={"ID":"6bd36ed1-3a87-445f-a966-77792741b82c","Type":"ContainerDied","Data":"3181598ee40e30f4ea2f279adcdd955bd72633dcc84770eb9480bf11be3d1037"}
Oct 30 14:45:02.708697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.708171    4530 scope.go:122] "RemoveContainer" containerID="3181598ee40e30f4ea2f279adcdd955bd72633dcc84770eb9480bf11be3d1037"
Oct 30 14:45:02.827637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:02.827587    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2" podStartSLOduration=2.8275704360000002 podStartE2EDuration="2.827570436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:02.82408567 +0000 UTC m=+3367.855984610" watchObservedRunningTime="2026-10-30 14:45:02.827570436 +0000 UTC m=+3367.859469376"
Oct 30 14:45:03.720206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:03.720171    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq" event={"ID":"6bd36ed1-3a87-445f-a966-77792741b82c","Type":"ContainerStarted","Data":"b0a48c9c9773d65e02648615ea6ec467c6b27e07e17b2e6589f6ca804f6326db"}
Oct 30 14:45:03.722427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:03.722403    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp" event={"ID":"b88f1bbb-88c1-4fff-9987-0e9ed0699865","Type":"ContainerStarted","Data":"a5a6876a96e7211a9d46a8afb12d630720f3570aa7b23c0cc96e966b2346eb3a"}
Oct 30 14:45:03.724488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:03.724461    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf" event={"ID":"24867fc7-ab36-491e-bccf-64a34aa01681","Type":"ContainerStarted","Data":"5bce4acb76cb5aab60d3bdefdc10cd52e40dd6248fb7f3c75ebe2869cf642e58"}
Oct 30 14:45:03.726497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:03.726472    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6" event={"ID":"7a608b93-87cb-4d9f-9d5a-8414d844b98e","Type":"ContainerStarted","Data":"eeb6f2bac28d666d9ae2850edca6bc366f881478d9c0750afaf057d5c3a3c8a4"}
Oct 30 14:45:03.728632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:03.728607    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5" event={"ID":"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5","Type":"ContainerStarted","Data":"166674fb38f5746c816377d672fed2055f6dda05377dd9fc4d049c05db6df4fc"}
Oct 30 14:45:03.730312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:03.730293    4530 generic.go:356] "Generic (PLEG): container finished" podID="0686b700-1a3d-4a49-82e0-73d5ac9de450" containerID="bd9d0a6888ab28a02c7328d408dfbbb8aba73bc96f051b381bfb685b447f9159" exitCode=60
Oct 30 14:45:03.730429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:03.730419    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2" event={"ID":"0686b700-1a3d-4a49-82e0-73d5ac9de450","Type":"ContainerDied","Data":"bd9d0a6888ab28a02c7328d408dfbbb8aba73bc96f051b381bfb685b447f9159"}
Oct 30 14:45:03.730753 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:03.730739    4530 scope.go:122] "RemoveContainer" containerID="bd9d0a6888ab28a02c7328d408dfbbb8aba73bc96f051b381bfb685b447f9159"
Oct 30 14:45:03.766431 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:03.766386    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq" podStartSLOduration=3.766373941 podStartE2EDuration="3.766373941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:03.765203217 +0000 UTC m=+3368.797102153" watchObservedRunningTime="2026-10-30 14:45:03.766373941 +0000 UTC m=+3368.798272880"
Oct 30 14:45:03.845563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:03.841215    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6" podStartSLOduration=3.8411963 podStartE2EDuration="3.8411963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:03.828107901 +0000 UTC m=+3368.860006844" watchObservedRunningTime="2026-10-30 14:45:03.8411963 +0000 UTC m=+3368.873095240"
Oct 30 14:45:03.927603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:03.926200    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp" podStartSLOduration=3.926179448 podStartE2EDuration="3.926179448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:03.920319032 +0000 UTC m=+3368.952217971" watchObservedRunningTime="2026-10-30 14:45:03.926179448 +0000 UTC m=+3368.958078388"
Oct 30 14:45:03.968843 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:03.968793    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf" podStartSLOduration=3.968778677 podStartE2EDuration="3.968778677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:03.965030828 +0000 UTC m=+3368.996929767" watchObservedRunningTime="2026-10-30 14:45:03.968778677 +0000 UTC m=+3369.000677616"
Oct 30 14:45:04.248795 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.248768    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:45:04.327667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.327614    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-certs\") pod \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") "
Oct 30 14:45:04.327877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.327722    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-socket\") pod \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") "
Oct 30 14:45:04.327971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.327948    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-envoy\") pod \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") "
Oct 30 14:45:04.328033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.328000    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-podinfo\") pod \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") "
Oct 30 14:45:04.328401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.328078    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-token\" (UniqueName: \"kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-token\") pod \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") "
Oct 30 14:45:04.328401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.328192    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-credential-socket\") pod \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") "
Oct 30 14:45:04.328401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.328229    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-data\") pod \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") "
Oct 30 14:45:04.328401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.328259    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-kube-api-access-d8mxm\" (UniqueName: \"kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-kube-api-access-d8mxm\") pod \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") "
Oct 30 14:45:04.328401 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.328315    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istiod-ca-cert\") pod \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\" (UID: \"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4\") "
Oct 30 14:45:04.332137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.328455    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-socket" pod "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" (UID: "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:04.332137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.328746    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-data" pod "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" (UID: "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:04.332137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.329003    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-credential-socket" pod "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" (UID: "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:04.332137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.329100    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istiod-ca-cert" pod "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" (UID: "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:45:04.332137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.329107    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:04.332137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.329145    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:04.332137 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.329353    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-certs" pod "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" (UID: "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:04.332534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.332318    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-envoy" pod "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" (UID: "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:04.332621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.332533    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-token" pod "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" (UID: "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:04.332769 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.332742    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-podinfo" pod "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" (UID: "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:45:04.334024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.333997    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-kube-api-access-d8mxm" pod "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" (UID: "0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4"). InnerVolumeSpecName "kube-api-access-d8mxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:04.430425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.430395    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:04.430425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.430422    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:04.430599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.430432    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:04.430599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.430443    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8mxm\" (UniqueName: \"kubernetes.io/projected/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-kube-api-access-d8mxm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:04.430599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.430466    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:04.430599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.430475    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:04.430599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.430483    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:04.746412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.746302    4530 generic.go:356] "Generic (PLEG): container finished" podID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" containerID="a5a6876a96e7211a9d46a8afb12d630720f3570aa7b23c0cc96e966b2346eb3a" exitCode=60
Oct 30 14:45:04.746412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.746389    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp" event={"ID":"b88f1bbb-88c1-4fff-9987-0e9ed0699865","Type":"ContainerDied","Data":"a5a6876a96e7211a9d46a8afb12d630720f3570aa7b23c0cc96e966b2346eb3a"}
Oct 30 14:45:04.746881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.746455    4530 scope.go:122] "RemoveContainer" containerID="300dbff144db9b9d33075ab2f4690352433e5a5f202c3d024f35a1ac0d342e92"
Oct 30 14:45:04.747303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.747285    4530 scope.go:122] "RemoveContainer" containerID="a5a6876a96e7211a9d46a8afb12d630720f3570aa7b23c0cc96e966b2346eb3a"
Oct 30 14:45:04.747972 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:04.747952    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp_odh-ai-gateway-infra(b88f1bbb-88c1-4fff-9987-0e9ed0699865)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865"
Oct 30 14:45:04.749283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.749258    4530 generic.go:356] "Generic (PLEG): container finished" podID="24867fc7-ab36-491e-bccf-64a34aa01681" containerID="5bce4acb76cb5aab60d3bdefdc10cd52e40dd6248fb7f3c75ebe2869cf642e58" exitCode=60
Oct 30 14:45:04.749402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.749343    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf" event={"ID":"24867fc7-ab36-491e-bccf-64a34aa01681","Type":"ContainerDied","Data":"5bce4acb76cb5aab60d3bdefdc10cd52e40dd6248fb7f3c75ebe2869cf642e58"}
Oct 30 14:45:04.749647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.749632    4530 scope.go:122] "RemoveContainer" containerID="5bce4acb76cb5aab60d3bdefdc10cd52e40dd6248fb7f3c75ebe2869cf642e58"
Oct 30 14:45:04.749998 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:04.749980    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf_odh-ai-gateway-infra(24867fc7-ab36-491e-bccf-64a34aa01681)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf" podUID="24867fc7-ab36-491e-bccf-64a34aa01681"
Oct 30 14:45:04.751941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.751918    4530 generic.go:356] "Generic (PLEG): container finished" podID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" containerID="eeb6f2bac28d666d9ae2850edca6bc366f881478d9c0750afaf057d5c3a3c8a4" exitCode=60
Oct 30 14:45:04.752071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.752054    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6" event={"ID":"7a608b93-87cb-4d9f-9d5a-8414d844b98e","Type":"ContainerDied","Data":"eeb6f2bac28d666d9ae2850edca6bc366f881478d9c0750afaf057d5c3a3c8a4"}
Oct 30 14:45:04.752245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.752230    4530 scope.go:122] "RemoveContainer" containerID="eeb6f2bac28d666d9ae2850edca6bc366f881478d9c0750afaf057d5c3a3c8a4"
Oct 30 14:45:04.752487 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:04.752466    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6_odh-ai-gateway-infra(7a608b93-87cb-4d9f-9d5a-8414d844b98e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e"
Oct 30 14:45:04.753983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.753961    4530 generic.go:356] "Generic (PLEG): container finished" podID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" containerID="166674fb38f5746c816377d672fed2055f6dda05377dd9fc4d049c05db6df4fc" exitCode=60
Oct 30 14:45:04.754065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.754014    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5" event={"ID":"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5","Type":"ContainerDied","Data":"166674fb38f5746c816377d672fed2055f6dda05377dd9fc4d049c05db6df4fc"}
Oct 30 14:45:04.754307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.754281    4530 scope.go:122] "RemoveContainer" containerID="166674fb38f5746c816377d672fed2055f6dda05377dd9fc4d049c05db6df4fc"
Oct 30 14:45:04.754590 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:04.754530    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5_odh-ai-gateway-infra(7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5"
Oct 30 14:45:04.756032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.755831    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2" event={"ID":"0686b700-1a3d-4a49-82e0-73d5ac9de450","Type":"ContainerStarted","Data":"56b1fc2d7ab2d814202eec24775d079c92ae7d2b773aaedebb0332970c2090e0"}
Oct 30 14:45:04.758201 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.758178    4530 generic.go:356] "Generic (PLEG): container finished" podID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerID="36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685" exitCode=0
Oct 30 14:45:04.758407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.758391    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"
Oct 30 14:45:04.759071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.759055    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" event={"ID":"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4","Type":"ContainerDied","Data":"36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685"}
Oct 30 14:45:04.759180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.759168    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b" event={"ID":"0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4","Type":"ContainerDied","Data":"5ed1104ddf16d2b8ce750336cbbd88ea88912449aa32d4911f5e532b6b0a8bc0"}
Oct 30 14:45:04.761019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.760993    4530 generic.go:356] "Generic (PLEG): container finished" podID="6bd36ed1-3a87-445f-a966-77792741b82c" containerID="b0a48c9c9773d65e02648615ea6ec467c6b27e07e17b2e6589f6ca804f6326db" exitCode=60
Oct 30 14:45:04.761125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.761073    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq" event={"ID":"6bd36ed1-3a87-445f-a966-77792741b82c","Type":"ContainerDied","Data":"b0a48c9c9773d65e02648615ea6ec467c6b27e07e17b2e6589f6ca804f6326db"}
Oct 30 14:45:04.761471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.761457    4530 scope.go:122] "RemoveContainer" containerID="b0a48c9c9773d65e02648615ea6ec467c6b27e07e17b2e6589f6ca804f6326db"
Oct 30 14:45:04.761795 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:04.761778    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq_odh-ai-gateway-infra(6bd36ed1-3a87-445f-a966-77792741b82c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq" podUID="6bd36ed1-3a87-445f-a966-77792741b82c"
Oct 30 14:45:04.764279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.764251    4530 scope.go:122] "RemoveContainer" containerID="5b545e54fe48c9aa82a9e4564d361c453c15c15245906bf8e4313d655b886524"
Oct 30 14:45:04.813370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.808141    4530 scope.go:122] "RemoveContainer" containerID="4f2012cfcf7992e34ac5bbc3383bd9726e498e430212152967d1530f1a2ed1f7"
Oct 30 14:45:04.827677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.827644    4530 scope.go:122] "RemoveContainer" containerID="f33c9fbf95062a251c99c37fca40aad85441b7e097976abf97e6f37954b18b3d"
Oct 30 14:45:04.845467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.845441    4530 scope.go:122] "RemoveContainer" containerID="36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685"
Oct 30 14:45:04.857954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.857924    4530 scope.go:122] "RemoveContainer" containerID="2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58"
Oct 30 14:45:04.869868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.869841    4530 scope.go:122] "RemoveContainer" containerID="36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685"
Oct 30 14:45:04.870348 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:04.870314    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685\": container with ID starting with 36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685 not found: ID does not exist" containerID="36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685"
Oct 30 14:45:04.870448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.870358    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685"} err="failed to get container status \"36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685\": rpc error: code = NotFound desc = could not find container \"36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685\": container with ID starting with 36589ea07529421391a0913d980030677b6cf0bc181a6b7bca451a17b04f5685 not found: ID does not exist"
Oct 30 14:45:04.870448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.870383    4530 scope.go:122] "RemoveContainer" containerID="2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58"
Oct 30 14:45:04.870824 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:04.870797    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58\": container with ID starting with 2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58 not found: ID does not exist" containerID="2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58"
Oct 30 14:45:04.870942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.870833    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58"} err="failed to get container status \"2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58\": rpc error: code = NotFound desc = could not find container \"2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58\": container with ID starting with 2db916c8f4ad20b1fa76b125c7ba8d3d2a7759a83d846c2d0c66dda7da418f58 not found: ID does not exist"
Oct 30 14:45:04.870942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.870859    4530 scope.go:122] "RemoveContainer" containerID="3181598ee40e30f4ea2f279adcdd955bd72633dcc84770eb9480bf11be3d1037"
Oct 30 14:45:04.948190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.948159    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"]
Oct 30 14:45:04.954359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:04.954329    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-vr67b"]
Oct 30 14:45:05.120092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:05.120020    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" path="/var/lib/kubelet/pods/0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4/volumes"
Oct 30 14:45:05.771154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:05.771108    4530 generic.go:356] "Generic (PLEG): container finished" podID="0686b700-1a3d-4a49-82e0-73d5ac9de450" containerID="56b1fc2d7ab2d814202eec24775d079c92ae7d2b773aaedebb0332970c2090e0" exitCode=60
Oct 30 14:45:05.771673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:05.771228    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2" event={"ID":"0686b700-1a3d-4a49-82e0-73d5ac9de450","Type":"ContainerDied","Data":"56b1fc2d7ab2d814202eec24775d079c92ae7d2b773aaedebb0332970c2090e0"}
Oct 30 14:45:05.771673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:05.771317    4530 scope.go:122] "RemoveContainer" containerID="bd9d0a6888ab28a02c7328d408dfbbb8aba73bc96f051b381bfb685b447f9159"
Oct 30 14:45:05.771819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:05.771791    4530 scope.go:122] "RemoveContainer" containerID="56b1fc2d7ab2d814202eec24775d079c92ae7d2b773aaedebb0332970c2090e0"
Oct 30 14:45:05.772216 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:05.772190    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2_odh-ai-gateway-infra(0686b700-1a3d-4a49-82e0-73d5ac9de450)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450"
Oct 30 14:45:05.776260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:05.776236    4530 scope.go:122] "RemoveContainer" containerID="b0a48c9c9773d65e02648615ea6ec467c6b27e07e17b2e6589f6ca804f6326db"
Oct 30 14:45:05.776677 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:05.776650    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq_odh-ai-gateway-infra(6bd36ed1-3a87-445f-a966-77792741b82c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq" podUID="6bd36ed1-3a87-445f-a966-77792741b82c"
Oct 30 14:45:05.778217 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:05.778174    4530 scope.go:122] "RemoveContainer" containerID="a5a6876a96e7211a9d46a8afb12d630720f3570aa7b23c0cc96e966b2346eb3a"
Oct 30 14:45:05.778492 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:05.778450    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp_odh-ai-gateway-infra(b88f1bbb-88c1-4fff-9987-0e9ed0699865)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865"
Oct 30 14:45:05.806588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:05.779780    4530 scope.go:122] "RemoveContainer" containerID="5bce4acb76cb5aab60d3bdefdc10cd52e40dd6248fb7f3c75ebe2869cf642e58"
Oct 30 14:45:05.806588 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:05.780037    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf_odh-ai-gateway-infra(24867fc7-ab36-491e-bccf-64a34aa01681)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf" podUID="24867fc7-ab36-491e-bccf-64a34aa01681"
Oct 30 14:45:05.811591 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:05.809357    4530 scope.go:122] "RemoveContainer" containerID="eeb6f2bac28d666d9ae2850edca6bc366f881478d9c0750afaf057d5c3a3c8a4"
Oct 30 14:45:05.811591 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:05.809740    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6_odh-ai-gateway-infra(7a608b93-87cb-4d9f-9d5a-8414d844b98e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e"
Oct 30 14:45:05.812420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:05.812390    4530 scope.go:122] "RemoveContainer" containerID="166674fb38f5746c816377d672fed2055f6dda05377dd9fc4d049c05db6df4fc"
Oct 30 14:45:05.812750 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:05.812724    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5_odh-ai-gateway-infra(7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5"
Oct 30 14:45:06.575854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.575815    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"]
Oct 30 14:45:06.576659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.576642    4530 cpu_manager.go:414] "containerMap: removing container" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy"
Oct 30 14:45:06.576739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.576660    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy"
Oct 30 14:45:06.576739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.576693    4530 cpu_manager.go:414] "containerMap: removing container" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy"
Oct 30 14:45:06.576739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.576698    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy"
Oct 30 14:45:06.576857 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.576846    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy"
Oct 30 14:45:06.602570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.591267    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.616805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.616775    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"]
Oct 30 14:45:06.671262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.671231    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.671462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.671391    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.671462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.671424    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.671645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.671463    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4h5c\" (UniqueName: \"kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-kube-api-access-f4h5c\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.671645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.671614    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.671762 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.671670    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.671819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.671761    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.671913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.671892    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.671974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.671956    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34398947-0d72-4b23-b8f8-e61dd6956a1a-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.773124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.773078    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.773124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.773136    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34398947-0d72-4b23-b8f8-e61dd6956a1a-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.773599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.773175    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.773599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.773198    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.773599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.773214    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.773599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.773238    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4h5c\" (UniqueName: \"kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-kube-api-access-f4h5c\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.773599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.773321    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.773599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.773352    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.773599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.773406    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.774338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.774307    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.774474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.774373    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.774474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.774400    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.774630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.774479    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.775088 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.775062    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34398947-0d72-4b23-b8f8-e61dd6956a1a-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.776087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.776062    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.776317 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.776299    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.856653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.852513    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4h5c\" (UniqueName: \"kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-kube-api-access-f4h5c\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.856653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.853837    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.934192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.934012    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:06.952597 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:06.943941    4530 scope.go:122] "RemoveContainer" containerID="56b1fc2d7ab2d814202eec24775d079c92ae7d2b773aaedebb0332970c2090e0"
Oct 30 14:45:06.952597 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:06.944392    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2_odh-ai-gateway-infra(0686b700-1a3d-4a49-82e0-73d5ac9de450)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450"
Oct 30 14:45:07.389789 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:45:07.389757    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34398947_0d72_4b23_b8f8_e61dd6956a1a.slice/crio-6870a1c52fa6e529475704de69b00a34ee94078dcc3f6938433abcee1dea81d7 WatchSource:0}: Error finding container 6870a1c52fa6e529475704de69b00a34ee94078dcc3f6938433abcee1dea81d7: Status 404 returned error can't find the container with id 6870a1c52fa6e529475704de69b00a34ee94078dcc3f6938433abcee1dea81d7
Oct 30 14:45:07.392893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:07.392846    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:45:07.392989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:07.392936    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:45:07.392989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:07.392966    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:45:07.394502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:07.394469    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"]
Oct 30 14:45:07.958563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:07.957981    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh" event={"ID":"34398947-0d72-4b23-b8f8-e61dd6956a1a","Type":"ContainerStarted","Data":"afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4"}
Oct 30 14:45:07.958563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:07.958077    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh" event={"ID":"34398947-0d72-4b23-b8f8-e61dd6956a1a","Type":"ContainerStarted","Data":"6870a1c52fa6e529475704de69b00a34ee94078dcc3f6938433abcee1dea81d7"}
Oct 30 14:45:08.011434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:08.011369    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh" podStartSLOduration=2.011348268 podStartE2EDuration="2.011348268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:08.011164206 +0000 UTC m=+3373.043063146" watchObservedRunningTime="2026-10-30 14:45:08.011348268 +0000 UTC m=+3373.043247208"
Oct 30 14:45:08.936387 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:08.936342    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:08.938026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:08.937994    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Oct 30 14:45:08.938171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:08.938047    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh" podUID="34398947-0d72-4b23-b8f8-e61dd6956a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Oct 30 14:45:09.409599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.409205    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"]
Oct 30 14:45:09.410251 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.410229    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebfd942-253e-4b76-aeb4-6b5bcd0b40d4" containerName="istio-proxy"
Oct 30 14:45:09.414448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.414424    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:09.427467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.427432    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"]
Oct 30 14:45:09.434967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.434936    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:45:09.438195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.438169    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj\" (UID: \"bf3b7066-08b0-4504-a19f-e66982a945df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:09.438511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.438490    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rpkf\" (UniqueName: \"kubernetes.io/projected/bf3b7066-08b0-4504-a19f-e66982a945df-kube-api-access-2rpkf\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj\" (UID: \"bf3b7066-08b0-4504-a19f-e66982a945df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:09.438835 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.438816    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj\" (UID: \"bf3b7066-08b0-4504-a19f-e66982a945df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:09.540821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.540769    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj\" (UID: \"bf3b7066-08b0-4504-a19f-e66982a945df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:09.541022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.541003    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rpkf\" (UniqueName: \"kubernetes.io/projected/bf3b7066-08b0-4504-a19f-e66982a945df-kube-api-access-2rpkf\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj\" (UID: \"bf3b7066-08b0-4504-a19f-e66982a945df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:09.541215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.541193    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj\" (UID: \"bf3b7066-08b0-4504-a19f-e66982a945df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:09.541467 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.541444    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj\" (UID: \"bf3b7066-08b0-4504-a19f-e66982a945df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:09.543781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.543749    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj\" (UID: \"bf3b7066-08b0-4504-a19f-e66982a945df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:09.570799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.570768    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rpkf\" (UniqueName: \"kubernetes.io/projected/bf3b7066-08b0-4504-a19f-e66982a945df-kube-api-access-2rpkf\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj\" (UID: \"bf3b7066-08b0-4504-a19f-e66982a945df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:09.737532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.737442    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:09.936441 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.936404    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Oct 30 14:45:09.936802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:09.936765    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh" podUID="34398947-0d72-4b23-b8f8-e61dd6956a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Oct 30 14:45:10.111640 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:45:10.110876    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3b7066_08b0_4504_a19f_e66982a945df.slice/crio-3eb14b5bc7b0c37ec8207ee3b2651ac5ed9c421bb22d2b000ffb0ceee2afd928 WatchSource:0}: Error finding container 3eb14b5bc7b0c37ec8207ee3b2651ac5ed9c421bb22d2b000ffb0ceee2afd928: Status 404 returned error can't find the container with id 3eb14b5bc7b0c37ec8207ee3b2651ac5ed9c421bb22d2b000ffb0ceee2afd928
Oct 30 14:45:10.113712 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:10.113463    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"]
Oct 30 14:45:10.936420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:10.936336    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Oct 30 14:45:10.936420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:10.936408    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh" podUID="34398947-0d72-4b23-b8f8-e61dd6956a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Oct 30 14:45:11.015215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:11.015168    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj" event={"ID":"bf3b7066-08b0-4504-a19f-e66982a945df","Type":"ContainerStarted","Data":"8dcf31c33893f296e1b6e54373a112d5f29bd814bba0e9483bd48389b462f4aa"}
Oct 30 14:45:11.015488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:11.015231    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj" event={"ID":"bf3b7066-08b0-4504-a19f-e66982a945df","Type":"ContainerStarted","Data":"3eb14b5bc7b0c37ec8207ee3b2651ac5ed9c421bb22d2b000ffb0ceee2afd928"}
Oct 30 14:45:11.069198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:11.069083    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj" podStartSLOduration=2.069065147 podStartE2EDuration="2.069065147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:11.067147678 +0000 UTC m=+3376.099046616" watchObservedRunningTime="2026-10-30 14:45:11.069065147 +0000 UTC m=+3376.100964086"
Oct 30 14:45:11.370155 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:11.370117    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:45:11.936369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:11.936324    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Oct 30 14:45:11.936523 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:11.936397    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh" podUID="34398947-0d72-4b23-b8f8-e61dd6956a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Oct 30 14:45:12.028838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:12.028801    4530 generic.go:356] "Generic (PLEG): container finished" podID="bf3b7066-08b0-4504-a19f-e66982a945df" containerID="8dcf31c33893f296e1b6e54373a112d5f29bd814bba0e9483bd48389b462f4aa" exitCode=0
Oct 30 14:45:12.029189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:12.028898    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj" event={"ID":"bf3b7066-08b0-4504-a19f-e66982a945df","Type":"ContainerDied","Data":"8dcf31c33893f296e1b6e54373a112d5f29bd814bba0e9483bd48389b462f4aa"}
Oct 30 14:45:12.573962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:12.573288    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": read tcp 10.133.0.2:51786->10.133.1.40:15021: read: connection reset by peer" start-of-body=
Oct 30 14:45:12.573962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:12.573351    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": read tcp 10.133.0.2:51786->10.133.1.40:15021: read: connection reset by peer"
Oct 30 14:45:12.936500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:12.936410    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Oct 30 14:45:12.936500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:12.936471    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh" podUID="34398947-0d72-4b23-b8f8-e61dd6956a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Oct 30 14:45:13.040462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.040428    4530 generic.go:356] "Generic (PLEG): container finished" podID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerID="2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9" exitCode=0
Oct 30 14:45:13.040692 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.040499    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" event={"ID":"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4","Type":"ContainerDied","Data":"2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9"}
Oct 30 14:45:13.041009 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.040986    4530 scope.go:122] "RemoveContainer" containerID="2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9"
Oct 30 14:45:13.043011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.042976    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:45:13.043127 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.043066    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:45:13.043127 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.043119    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:45:13.351605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.351578    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:13.443728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.443694    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3b7066-08b0-4504-a19f-e66982a945df-kube-api-access-2rpkf\" (UniqueName: \"kubernetes.io/projected/bf3b7066-08b0-4504-a19f-e66982a945df-kube-api-access-2rpkf\") pod \"bf3b7066-08b0-4504-a19f-e66982a945df\" (UID: \"bf3b7066-08b0-4504-a19f-e66982a945df\") "
Oct 30 14:45:13.443947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.443901    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-xks-ca\") pod \"bf3b7066-08b0-4504-a19f-e66982a945df\" (UID: \"bf3b7066-08b0-4504-a19f-e66982a945df\") "
Oct 30 14:45:13.444027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.443958    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-openshift-ca\") pod \"bf3b7066-08b0-4504-a19f-e66982a945df\" (UID: \"bf3b7066-08b0-4504-a19f-e66982a945df\") "
Oct 30 14:45:13.445032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.444997    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-openshift-ca" pod "bf3b7066-08b0-4504-a19f-e66982a945df" (UID: "bf3b7066-08b0-4504-a19f-e66982a945df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:45:13.447504 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.447477    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3b7066-08b0-4504-a19f-e66982a945df-kube-api-access-2rpkf" pod "bf3b7066-08b0-4504-a19f-e66982a945df" (UID: "bf3b7066-08b0-4504-a19f-e66982a945df"). InnerVolumeSpecName "kube-api-access-2rpkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:13.447719 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.447480    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-xks-ca" pod "bf3b7066-08b0-4504-a19f-e66982a945df" (UID: "bf3b7066-08b0-4504-a19f-e66982a945df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:45:13.554999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.554951    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rpkf\" (UniqueName: \"kubernetes.io/projected/bf3b7066-08b0-4504-a19f-e66982a945df-kube-api-access-2rpkf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:13.554999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.555002    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:13.555292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.555021    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3b7066-08b0-4504-a19f-e66982a945df-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:13.941617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.941579    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:13.942063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.942043    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:13.942807 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:13.942787    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:45:14.077039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:14.076994    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" event={"ID":"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4","Type":"ContainerStarted","Data":"16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05"}
Oct 30 14:45:14.088392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:14.088354    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"
Oct 30 14:45:14.089130 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:14.089098    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj" event={"ID":"bf3b7066-08b0-4504-a19f-e66982a945df","Type":"ContainerDied","Data":"3eb14b5bc7b0c37ec8207ee3b2651ac5ed9c421bb22d2b000ffb0ceee2afd928"}
Oct 30 14:45:14.095560 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:14.092810    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eb14b5bc7b0c37ec8207ee3b2651ac5ed9c421bb22d2b000ffb0ceee2afd928"
Oct 30 14:45:14.237404 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:14.237309    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:45:14.237612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:14.237497    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 14:45:14.237612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:14.237579    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 14:45:15.118270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.118241    4530 scope.go:122] "RemoveContainer" containerID="166674fb38f5746c816377d672fed2055f6dda05377dd9fc4d049c05db6df4fc"
Oct 30 14:45:15.237568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.237513    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 14:45:15.237746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.237599    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 14:45:15.372605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.372487    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" containerID="cri-o://6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b" gracePeriod=45
Oct 30 14:45:15.434726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.434684    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": dial tcp 10.133.0.109:8000: connect: connection refused"
Oct 30 14:45:15.719876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.719848    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:45:15.819074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.819033    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-model-cache\") pod \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") "
Oct 30 14:45:15.819292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.819096    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-dshm\") pod \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") "
Oct 30 14:45:15.819292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.819212    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tmp-dir\") pod \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") "
Oct 30 14:45:15.819292 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.819269    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-home\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-home\") pod \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") "
Oct 30 14:45:15.819499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.819366    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-kube-api-access-n5vdm\" (UniqueName: \"kubernetes.io/projected/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-kube-api-access-n5vdm\") pod \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") "
Oct 30 14:45:15.819499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.819475    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tls-certs\") pod \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\" (UID: \"10b2c1ea-30bd-44be-baaf-5fe82ee558a5\") "
Oct 30 14:45:15.820765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.820732    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-home" pod "10b2c1ea-30bd-44be-baaf-5fe82ee558a5" (UID: "10b2c1ea-30bd-44be-baaf-5fe82ee558a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:15.821589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.821561    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-model-cache" pod "10b2c1ea-30bd-44be-baaf-5fe82ee558a5" (UID: "10b2c1ea-30bd-44be-baaf-5fe82ee558a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:15.821865 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.821799    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tmp-dir" pod "10b2c1ea-30bd-44be-baaf-5fe82ee558a5" (UID: "10b2c1ea-30bd-44be-baaf-5fe82ee558a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:15.824154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.824128    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tls-certs" pod "10b2c1ea-30bd-44be-baaf-5fe82ee558a5" (UID: "10b2c1ea-30bd-44be-baaf-5fe82ee558a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:45:15.828240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.828205    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-kube-api-access-n5vdm" pod "10b2c1ea-30bd-44be-baaf-5fe82ee558a5" (UID: "10b2c1ea-30bd-44be-baaf-5fe82ee558a5"). InnerVolumeSpecName "kube-api-access-n5vdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:15.828744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.828712    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-dshm" pod "10b2c1ea-30bd-44be-baaf-5fe82ee558a5" (UID: "10b2c1ea-30bd-44be-baaf-5fe82ee558a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:15.921810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.921725    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:15.921810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.921753    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:15.921810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.921765    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5vdm\" (UniqueName: \"kubernetes.io/projected/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-kube-api-access-n5vdm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:15.921810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.921773    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:15.921810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.921782    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:15.921810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:15.921789    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10b2c1ea-30bd-44be-baaf-5fe82ee558a5-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:16.115567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.115256    4530 scope.go:122] "RemoveContainer" containerID="b0a48c9c9773d65e02648615ea6ec467c6b27e07e17b2e6589f6ca804f6326db"
Oct 30 14:45:16.116377 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.116323    4530 scope.go:122] "RemoveContainer" containerID="5bce4acb76cb5aab60d3bdefdc10cd52e40dd6248fb7f3c75ebe2869cf642e58"
Oct 30 14:45:16.116756 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.116720    4530 scope.go:122] "RemoveContainer" containerID="56b1fc2d7ab2d814202eec24775d079c92ae7d2b773aaedebb0332970c2090e0"
Oct 30 14:45:16.127051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.127028    4530 generic.go:356] "Generic (PLEG): container finished" podID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerID="6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b" exitCode=0
Oct 30 14:45:16.127445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.127064    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"
Oct 30 14:45:16.127445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.127113    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" event={"ID":"10b2c1ea-30bd-44be-baaf-5fe82ee558a5","Type":"ContainerDied","Data":"6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b"}
Oct 30 14:45:16.127445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.127159    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2" event={"ID":"10b2c1ea-30bd-44be-baaf-5fe82ee558a5","Type":"ContainerDied","Data":"ea282c0f85f0e7eeb446a341e870a9dc27202e3aa6d35edda0cc2d792a05e6de"}
Oct 30 14:45:16.127445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.127186    4530 scope.go:122] "RemoveContainer" containerID="6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b"
Oct 30 14:45:16.132363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.132341    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5" event={"ID":"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5","Type":"ContainerStarted","Data":"be87a0e753928a66ffe40ad84b3026bc04cef361ed1ce7d802781ef24a85a273"}
Oct 30 14:45:16.152241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.152222    4530 scope.go:122] "RemoveContainer" containerID="6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b"
Oct 30 14:45:16.152555 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:16.152512    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b\": container with ID starting with 6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b not found: ID does not exist" containerID="6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b"
Oct 30 14:45:16.152661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.152585    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b"} err="failed to get container status \"6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b\": rpc error: code = NotFound desc = could not find container \"6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b\": container with ID starting with 6ca149d75bc7d0729d0127122ae64d2424b885dcfdac32a776456c54c732629b not found: ID does not exist"
Oct 30 14:45:16.235970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.235938    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"]
Oct 30 14:45:16.237516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.237488    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 14:45:16.237612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.237568    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 14:45:16.241104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:16.241082    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/distinct-2-w1-ca229c-kserve-769ffbd675-4kvc2"]
Oct 30 14:45:17.118482 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.118363    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" path="/var/lib/kubelet/pods/10b2c1ea-30bd-44be-baaf-5fe82ee558a5/volumes"
Oct 30 14:45:17.142890 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.142770    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf" event={"ID":"24867fc7-ab36-491e-bccf-64a34aa01681","Type":"ContainerStarted","Data":"991d4db64793cddf34bb91ddd54cf7b75f25d85c11a557674fb78b5e12ed03fc"}
Oct 30 14:45:17.149104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.149073    4530 generic.go:356] "Generic (PLEG): container finished" podID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" containerID="be87a0e753928a66ffe40ad84b3026bc04cef361ed1ce7d802781ef24a85a273" exitCode=60
Oct 30 14:45:17.149242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.149144    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5" event={"ID":"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5","Type":"ContainerDied","Data":"be87a0e753928a66ffe40ad84b3026bc04cef361ed1ce7d802781ef24a85a273"}
Oct 30 14:45:17.149242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.149201    4530 scope.go:122] "RemoveContainer" containerID="166674fb38f5746c816377d672fed2055f6dda05377dd9fc4d049c05db6df4fc"
Oct 30 14:45:17.149608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.149590    4530 scope.go:122] "RemoveContainer" containerID="be87a0e753928a66ffe40ad84b3026bc04cef361ed1ce7d802781ef24a85a273"
Oct 30 14:45:17.149928 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:17.149905    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5_odh-ai-gateway-infra(7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5"
Oct 30 14:45:17.151939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.151923    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"]
Oct 30 14:45:17.152897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.152866    4530 cpu_manager.go:414] "containerMap: removing container" podUID="bf3b7066-08b0-4504-a19f-e66982a945df" containerName="revoke-keys"
Oct 30 14:45:17.153002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.152897    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3b7066-08b0-4504-a19f-e66982a945df" containerName="revoke-keys"
Oct 30 14:45:17.153002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.152960    4530 cpu_manager.go:414] "containerMap: removing container" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main"
Oct 30 14:45:17.153002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.152967    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main"
Oct 30 14:45:17.153146 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.153125    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b2c1ea-30bd-44be-baaf-5fe82ee558a5" containerName="main"
Oct 30 14:45:17.153247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.153151    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3b7066-08b0-4504-a19f-e66982a945df" containerName="revoke-keys"
Oct 30 14:45:17.156166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.156145    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2" event={"ID":"0686b700-1a3d-4a49-82e0-73d5ac9de450","Type":"ContainerStarted","Data":"8f7408848ad169d494fa24261a7487047ffad11f4431bd09e8a25e02f9702d4b"}
Oct 30 14:45:17.156281 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.156173    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq" event={"ID":"6bd36ed1-3a87-445f-a966-77792741b82c","Type":"ContainerStarted","Data":"669368c3c2a2336082e9ab173b05f9e885d7e76b6ddf65ebb11d9c4a31d9fe89"}
Oct 30 14:45:17.156281 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.156269    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.176633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.175919    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-bcc710-serving-cert\""
Oct 30 14:45:17.213606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.213493    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"]
Oct 30 14:45:17.226268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.226233    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"]
Oct 30 14:45:17.242612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.242499    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 14:45:17.242776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.242608    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 14:45:17.243397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.243068    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-metrics-certs\") pod \"maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.243397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.243105    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.243397 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.243200    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93cd08a-fef5-441b-9898-799dba609917-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.244114 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.243989    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnnwl\" (UniqueName: \"kubernetes.io/projected/d93cd08a-fef5-441b-9898-799dba609917-kube-api-access-xnnwl\") pod \"maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.346100 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.346063    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93cd08a-fef5-441b-9898-799dba609917-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.346267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.346254    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnnwl\" (UniqueName: \"kubernetes.io/projected/d93cd08a-fef5-441b-9898-799dba609917-kube-api-access-xnnwl\") pod \"maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.346435 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.346410    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-metrics-certs\") pod \"maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.346507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.346462    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.347147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.347118    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93cd08a-fef5-441b-9898-799dba609917-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.349499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.349474    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-metrics-certs\") pod \"maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.350147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.350108    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.392778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.392417    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnnwl\" (UniqueName: \"kubernetes.io/projected/d93cd08a-fef5-441b-9898-799dba609917-kube-api-access-xnnwl\") pod \"maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.475229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.475197    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:17.818848 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:17.818773    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"]
Oct 30 14:45:18.108395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.108294    4530 scope.go:122] "RemoveContainer" containerID="eeb6f2bac28d666d9ae2850edca6bc366f881478d9c0750afaf057d5c3a3c8a4"
Oct 30 14:45:18.109726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.108473    4530 scope.go:122] "RemoveContainer" containerID="a5a6876a96e7211a9d46a8afb12d630720f3570aa7b23c0cc96e966b2346eb3a"
Oct 30 14:45:18.242698 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.242659    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 14:45:18.243200 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.242717    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 14:45:18.243352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.243332    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"]
Oct 30 14:45:18.250157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.250129    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4" event={"ID":"d93cd08a-fef5-441b-9898-799dba609917","Type":"ContainerStarted","Data":"f4106cbf5467ff4a030a19337665ca598720ab5013d6c6020252e522a3d6d9d4"}
Oct 30 14:45:18.252301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.252278    4530 generic.go:356] "Generic (PLEG): container finished" podID="0686b700-1a3d-4a49-82e0-73d5ac9de450" containerID="8f7408848ad169d494fa24261a7487047ffad11f4431bd09e8a25e02f9702d4b" exitCode=60
Oct 30 14:45:18.252408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.252371    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2" event={"ID":"0686b700-1a3d-4a49-82e0-73d5ac9de450","Type":"ContainerDied","Data":"8f7408848ad169d494fa24261a7487047ffad11f4431bd09e8a25e02f9702d4b"}
Oct 30 14:45:18.252455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.252411    4530 scope.go:122] "RemoveContainer" containerID="56b1fc2d7ab2d814202eec24775d079c92ae7d2b773aaedebb0332970c2090e0"
Oct 30 14:45:18.252902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.252884    4530 scope.go:122] "RemoveContainer" containerID="8f7408848ad169d494fa24261a7487047ffad11f4431bd09e8a25e02f9702d4b"
Oct 30 14:45:18.253324 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:18.253300    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2_odh-ai-gateway-infra(0686b700-1a3d-4a49-82e0-73d5ac9de450)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450"
Oct 30 14:45:18.254980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.254963    4530 generic.go:356] "Generic (PLEG): container finished" podID="6bd36ed1-3a87-445f-a966-77792741b82c" containerID="669368c3c2a2336082e9ab173b05f9e885d7e76b6ddf65ebb11d9c4a31d9fe89" exitCode=60
Oct 30 14:45:18.255151 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.255037    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq" event={"ID":"6bd36ed1-3a87-445f-a966-77792741b82c","Type":"ContainerDied","Data":"669368c3c2a2336082e9ab173b05f9e885d7e76b6ddf65ebb11d9c4a31d9fe89"}
Oct 30 14:45:18.255582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.255478    4530 scope.go:122] "RemoveContainer" containerID="669368c3c2a2336082e9ab173b05f9e885d7e76b6ddf65ebb11d9c4a31d9fe89"
Oct 30 14:45:18.255842 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:18.255816    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq_odh-ai-gateway-infra(6bd36ed1-3a87-445f-a966-77792741b82c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq" podUID="6bd36ed1-3a87-445f-a966-77792741b82c"
Oct 30 14:45:18.257481 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.257391    4530 generic.go:356] "Generic (PLEG): container finished" podID="24867fc7-ab36-491e-bccf-64a34aa01681" containerID="991d4db64793cddf34bb91ddd54cf7b75f25d85c11a557674fb78b5e12ed03fc" exitCode=60
Oct 30 14:45:18.257596 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.257471    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf" event={"ID":"24867fc7-ab36-491e-bccf-64a34aa01681","Type":"ContainerDied","Data":"991d4db64793cddf34bb91ddd54cf7b75f25d85c11a557674fb78b5e12ed03fc"}
Oct 30 14:45:18.269279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.269238    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"]
Oct 30 14:45:18.271498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.271482    4530 scope.go:122] "RemoveContainer" containerID="b0a48c9c9773d65e02648615ea6ec467c6b27e07e17b2e6589f6ca804f6326db"
Oct 30 14:45:18.322083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.322052    4530 scope.go:122] "RemoveContainer" containerID="5bce4acb76cb5aab60d3bdefdc10cd52e40dd6248fb7f3c75ebe2869cf642e58"
Oct 30 14:45:18.331475 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.331449    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"]
Oct 30 14:45:18.720200 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.710394    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"
Oct 30 14:45:18.773080 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.773039    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24867fc7-ab36-491e-bccf-64a34aa01681-kube-api-access-tjf9k\" (UniqueName: \"kubernetes.io/projected/24867fc7-ab36-491e-bccf-64a34aa01681-kube-api-access-tjf9k\") pod \"24867fc7-ab36-491e-bccf-64a34aa01681\" (UID: \"24867fc7-ab36-491e-bccf-64a34aa01681\") "
Oct 30 14:45:18.773380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.773356    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24867fc7-ab36-491e-bccf-64a34aa01681-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24867fc7-ab36-491e-bccf-64a34aa01681-maas-api-service-ca\") pod \"24867fc7-ab36-491e-bccf-64a34aa01681\" (UID: \"24867fc7-ab36-491e-bccf-64a34aa01681\") "
Oct 30 14:45:18.773878 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.773847    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24867fc7-ab36-491e-bccf-64a34aa01681-maas-api-service-ca" pod "24867fc7-ab36-491e-bccf-64a34aa01681" (UID: "24867fc7-ab36-491e-bccf-64a34aa01681"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:45:18.774095 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.774076    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24867fc7-ab36-491e-bccf-64a34aa01681-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:18.825337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.823128    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24867fc7-ab36-491e-bccf-64a34aa01681-kube-api-access-tjf9k" pod "24867fc7-ab36-491e-bccf-64a34aa01681" (UID: "24867fc7-ab36-491e-bccf-64a34aa01681"). InnerVolumeSpecName "kube-api-access-tjf9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:18.829082 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.828823    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"
Oct 30 14:45:18.875190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.875149    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-maas-api-service-ca\") pod \"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5\" (UID: \"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5\") "
Oct 30 14:45:18.875995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.875220    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-kube-api-access-67wmf\" (UniqueName: \"kubernetes.io/projected/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-kube-api-access-67wmf\") pod \"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5\" (UID: \"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5\") "
Oct 30 14:45:18.875995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.875711    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-maas-api-service-ca" pod "7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" (UID: "7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:45:18.876071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.875994    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:18.876071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.876011    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjf9k\" (UniqueName: \"kubernetes.io/projected/24867fc7-ab36-491e-bccf-64a34aa01681-kube-api-access-tjf9k\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:18.877522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.877494    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-kube-api-access-67wmf" pod "7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" (UID: "7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5"). InnerVolumeSpecName "kube-api-access-67wmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:18.977796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:18.977706    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67wmf\" (UniqueName: \"kubernetes.io/projected/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5-kube-api-access-67wmf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:19.237718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.237625    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 14:45:19.237718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.237690    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 14:45:19.371457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.371421    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp" event={"ID":"b88f1bbb-88c1-4fff-9987-0e9ed0699865","Type":"ContainerStarted","Data":"5f839323a66b735eb7356919879813597d633194940c1c0f38df4a061d34a7d6"}
Oct 30 14:45:19.411269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.411223    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf" event={"ID":"24867fc7-ab36-491e-bccf-64a34aa01681","Type":"ContainerDied","Data":"bdc2b03c79795b9c94d85bfcd25a382dd5c189b63205242634bcd987315c2e3e"}
Oct 30 14:45:19.411419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.411278    4530 scope.go:122] "RemoveContainer" containerID="991d4db64793cddf34bb91ddd54cf7b75f25d85c11a557674fb78b5e12ed03fc"
Oct 30 14:45:19.411419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.411357    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"
Oct 30 14:45:19.415124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.415062    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6" event={"ID":"7a608b93-87cb-4d9f-9d5a-8414d844b98e","Type":"ContainerStarted","Data":"2cc82b0b805c758917edfb72f2070cf17145960e13a9dd0fd5a7c543528b3cdd"}
Oct 30 14:45:19.419057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.418263    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5" event={"ID":"7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5","Type":"ContainerDied","Data":"6349c9faecd4f8144c1eef4cb63d44f264ed6fe6207b4c103ae431f84628b07d"}
Oct 30 14:45:19.419057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.418369    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"
Oct 30 14:45:19.423011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.422335    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4" event={"ID":"d93cd08a-fef5-441b-9898-799dba609917","Type":"ContainerStarted","Data":"26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba"}
Oct 30 14:45:19.423011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.422976    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:19.434187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.434170    4530 scope.go:122] "RemoveContainer" containerID="be87a0e753928a66ffe40ad84b3026bc04cef361ed1ce7d802781ef24a85a273"
Oct 30 14:45:19.606727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.606591    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"]
Oct 30 14:45:19.616586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.614629    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889525-r2nl5"]
Oct 30 14:45:19.663665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.662510    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4" podStartSLOduration=2.138748293 podStartE2EDuration="2.662495105s" totalImagesPullingTime="523.746812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:19.659321596 +0000 UTC m=+3384.691220536" watchObservedRunningTime="2026-10-30 14:45:19.662495105 +0000 UTC m=+3384.694394076"
Oct 30 14:45:19.674398 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.674370    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"
Oct 30 14:45:19.719448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.719408    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"
Oct 30 14:45:19.773727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.772213    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd36ed1-3a87-445f-a966-77792741b82c-kube-api-access-w4hmm\" (UniqueName: \"kubernetes.io/projected/6bd36ed1-3a87-445f-a966-77792741b82c-kube-api-access-w4hmm\") pod \"6bd36ed1-3a87-445f-a966-77792741b82c\" (UID: \"6bd36ed1-3a87-445f-a966-77792741b82c\") "
Oct 30 14:45:19.773727 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.772396    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd36ed1-3a87-445f-a966-77792741b82c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd36ed1-3a87-445f-a966-77792741b82c-maas-api-service-ca\") pod \"6bd36ed1-3a87-445f-a966-77792741b82c\" (UID: \"6bd36ed1-3a87-445f-a966-77792741b82c\") "
Oct 30 14:45:19.800015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.797767    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd36ed1-3a87-445f-a966-77792741b82c-maas-api-service-ca" pod "6bd36ed1-3a87-445f-a966-77792741b82c" (UID: "6bd36ed1-3a87-445f-a966-77792741b82c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:45:19.805892 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.805857    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd36ed1-3a87-445f-a966-77792741b82c-kube-api-access-w4hmm" pod "6bd36ed1-3a87-445f-a966-77792741b82c" (UID: "6bd36ed1-3a87-445f-a966-77792741b82c"). InnerVolumeSpecName "kube-api-access-w4hmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:19.833040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.833006    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"]
Oct 30 14:45:19.838964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.838930    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-90b406-29889525-b75gf"]
Oct 30 14:45:19.876125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.876086    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0686b700-1a3d-4a49-82e0-73d5ac9de450-kube-api-access-kmk5n\" (UniqueName: \"kubernetes.io/projected/0686b700-1a3d-4a49-82e0-73d5ac9de450-kube-api-access-kmk5n\") pod \"0686b700-1a3d-4a49-82e0-73d5ac9de450\" (UID: \"0686b700-1a3d-4a49-82e0-73d5ac9de450\") "
Oct 30 14:45:19.876780 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.876750    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0686b700-1a3d-4a49-82e0-73d5ac9de450-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0686b700-1a3d-4a49-82e0-73d5ac9de450-maas-api-service-ca\") pod \"0686b700-1a3d-4a49-82e0-73d5ac9de450\" (UID: \"0686b700-1a3d-4a49-82e0-73d5ac9de450\") "
Oct 30 14:45:19.878031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.877865    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4hmm\" (UniqueName: \"kubernetes.io/projected/6bd36ed1-3a87-445f-a966-77792741b82c-kube-api-access-w4hmm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:19.878252 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.878234    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd36ed1-3a87-445f-a966-77792741b82c-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:19.878837 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.878812    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0686b700-1a3d-4a49-82e0-73d5ac9de450-maas-api-service-ca" pod "0686b700-1a3d-4a49-82e0-73d5ac9de450" (UID: "0686b700-1a3d-4a49-82e0-73d5ac9de450"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:45:19.881268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.881235    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0686b700-1a3d-4a49-82e0-73d5ac9de450-kube-api-access-kmk5n" pod "0686b700-1a3d-4a49-82e0-73d5ac9de450" (UID: "0686b700-1a3d-4a49-82e0-73d5ac9de450"). InnerVolumeSpecName "kube-api-access-kmk5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:19.980294 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.980247    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmk5n\" (UniqueName: \"kubernetes.io/projected/0686b700-1a3d-4a49-82e0-73d5ac9de450-kube-api-access-kmk5n\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:19.980294 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:19.980288    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0686b700-1a3d-4a49-82e0-73d5ac9de450-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:20.241828 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.241794    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:45:20.242301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.242287    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:45:20.243003 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.242983    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:45:20.433495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.433458    4530 generic.go:356] "Generic (PLEG): container finished" podID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" containerID="5f839323a66b735eb7356919879813597d633194940c1c0f38df4a061d34a7d6" exitCode=60
Oct 30 14:45:20.434002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.433533    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp" event={"ID":"b88f1bbb-88c1-4fff-9987-0e9ed0699865","Type":"ContainerDied","Data":"5f839323a66b735eb7356919879813597d633194940c1c0f38df4a061d34a7d6"}
Oct 30 14:45:20.434002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.433642    4530 scope.go:122] "RemoveContainer" containerID="a5a6876a96e7211a9d46a8afb12d630720f3570aa7b23c0cc96e966b2346eb3a"
Oct 30 14:45:20.434170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.434133    4530 scope.go:122] "RemoveContainer" containerID="5f839323a66b735eb7356919879813597d633194940c1c0f38df4a061d34a7d6"
Oct 30 14:45:20.434601 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:20.434572    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp_odh-ai-gateway-infra(b88f1bbb-88c1-4fff-9987-0e9ed0699865)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865"
Oct 30 14:45:20.437629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.437599    4530 generic.go:356] "Generic (PLEG): container finished" podID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" containerID="2cc82b0b805c758917edfb72f2070cf17145960e13a9dd0fd5a7c543528b3cdd" exitCode=60
Oct 30 14:45:20.437745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.437674    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6" event={"ID":"7a608b93-87cb-4d9f-9d5a-8414d844b98e","Type":"ContainerDied","Data":"2cc82b0b805c758917edfb72f2070cf17145960e13a9dd0fd5a7c543528b3cdd"}
Oct 30 14:45:20.438079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.438063    4530 scope.go:122] "RemoveContainer" containerID="2cc82b0b805c758917edfb72f2070cf17145960e13a9dd0fd5a7c543528b3cdd"
Oct 30 14:45:20.438445 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:20.438418    4530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6_odh-ai-gateway-infra(7a608b93-87cb-4d9f-9d5a-8414d844b98e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e"
Oct 30 14:45:20.440676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.440650    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2" event={"ID":"0686b700-1a3d-4a49-82e0-73d5ac9de450","Type":"ContainerDied","Data":"534cabb624b180234dc4893c0fc8da985924fa59d485bc345fd4bc8434d6ee0f"}
Oct 30 14:45:20.440793 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.440690    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"
Oct 30 14:45:20.442295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.442269    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq" event={"ID":"6bd36ed1-3a87-445f-a966-77792741b82c","Type":"ContainerDied","Data":"eac8d04963d55c559858ca03d937f9e53eaf522aea995c5c9b2d5d03086b48a8"}
Oct 30 14:45:20.442405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.442353    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"
Oct 30 14:45:20.448981 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.448962    4530 scope.go:122] "RemoveContainer" containerID="eeb6f2bac28d666d9ae2850edca6bc366f881478d9c0750afaf057d5c3a3c8a4"
Oct 30 14:45:20.463296 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.463279    4530 scope.go:122] "RemoveContainer" containerID="8f7408848ad169d494fa24261a7487047ffad11f4431bd09e8a25e02f9702d4b"
Oct 30 14:45:20.472852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.472832    4530 scope.go:122] "RemoveContainer" containerID="669368c3c2a2336082e9ab173b05f9e885d7e76b6ddf65ebb11d9c4a31d9fe89"
Oct 30 14:45:20.531980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.531898    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"]
Oct 30 14:45:20.536458 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.536430    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-75443e-29889525-6x7m2"]
Oct 30 14:45:20.580903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.580742    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"]
Oct 30 14:45:20.663071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.662982    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6176c3-29889525-9x9xq"]
Oct 30 14:45:20.729129 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.729089    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"]
Oct 30 14:45:20.774610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:20.774570    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"]
Oct 30 14:45:21.116436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.116405    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450" path="/var/lib/kubelet/pods/0686b700-1a3d-4a49-82e0-73d5ac9de450/volumes"
Oct 30 14:45:21.116957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.116939    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24867fc7-ab36-491e-bccf-64a34aa01681" path="/var/lib/kubelet/pods/24867fc7-ab36-491e-bccf-64a34aa01681/volumes"
Oct 30 14:45:21.117351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.117336    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd36ed1-3a87-445f-a966-77792741b82c" path="/var/lib/kubelet/pods/6bd36ed1-3a87-445f-a966-77792741b82c/volumes"
Oct 30 14:45:21.118219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.118201    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" path="/var/lib/kubelet/pods/7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5/volumes"
Oct 30 14:45:21.726083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.725651    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"
Oct 30 14:45:21.734405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.734331    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"
Oct 30 14:45:21.815240 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.815198    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88f1bbb-88c1-4fff-9987-0e9ed0699865-kube-api-access-j9827\" (UniqueName: \"kubernetes.io/projected/b88f1bbb-88c1-4fff-9987-0e9ed0699865-kube-api-access-j9827\") pod \"b88f1bbb-88c1-4fff-9987-0e9ed0699865\" (UID: \"b88f1bbb-88c1-4fff-9987-0e9ed0699865\") "
Oct 30 14:45:21.815430 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.815276    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a608b93-87cb-4d9f-9d5a-8414d844b98e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a608b93-87cb-4d9f-9d5a-8414d844b98e-maas-api-service-ca\") pod \"7a608b93-87cb-4d9f-9d5a-8414d844b98e\" (UID: \"7a608b93-87cb-4d9f-9d5a-8414d844b98e\") "
Oct 30 14:45:21.815748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.815716    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a608b93-87cb-4d9f-9d5a-8414d844b98e-maas-api-service-ca" pod "7a608b93-87cb-4d9f-9d5a-8414d844b98e" (UID: "7a608b93-87cb-4d9f-9d5a-8414d844b98e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:45:21.815885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.815862    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88f1bbb-88c1-4fff-9987-0e9ed0699865-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88f1bbb-88c1-4fff-9987-0e9ed0699865-maas-api-service-ca\") pod \"b88f1bbb-88c1-4fff-9987-0e9ed0699865\" (UID: \"b88f1bbb-88c1-4fff-9987-0e9ed0699865\") "
Oct 30 14:45:21.816328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.816303    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88f1bbb-88c1-4fff-9987-0e9ed0699865-maas-api-service-ca" pod "b88f1bbb-88c1-4fff-9987-0e9ed0699865" (UID: "b88f1bbb-88c1-4fff-9987-0e9ed0699865"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:45:21.816449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.816372    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a608b93-87cb-4d9f-9d5a-8414d844b98e-kube-api-access-vt4zc\" (UniqueName: \"kubernetes.io/projected/7a608b93-87cb-4d9f-9d5a-8414d844b98e-kube-api-access-vt4zc\") pod \"7a608b93-87cb-4d9f-9d5a-8414d844b98e\" (UID: \"7a608b93-87cb-4d9f-9d5a-8414d844b98e\") "
Oct 30 14:45:21.817876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.817507    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a608b93-87cb-4d9f-9d5a-8414d844b98e-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:21.817876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.817535    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88f1bbb-88c1-4fff-9987-0e9ed0699865-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:21.819051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.819024    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88f1bbb-88c1-4fff-9987-0e9ed0699865-kube-api-access-j9827" pod "b88f1bbb-88c1-4fff-9987-0e9ed0699865" (UID: "b88f1bbb-88c1-4fff-9987-0e9ed0699865"). InnerVolumeSpecName "kube-api-access-j9827". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:21.819295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.819270    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a608b93-87cb-4d9f-9d5a-8414d844b98e-kube-api-access-vt4zc" pod "7a608b93-87cb-4d9f-9d5a-8414d844b98e" (UID: "7a608b93-87cb-4d9f-9d5a-8414d844b98e"). InnerVolumeSpecName "kube-api-access-vt4zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:21.919162 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.919125    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt4zc\" (UniqueName: \"kubernetes.io/projected/7a608b93-87cb-4d9f-9d5a-8414d844b98e-kube-api-access-vt4zc\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:21.919162 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:21.919161    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9827\" (UniqueName: \"kubernetes.io/projected/b88f1bbb-88c1-4fff-9987-0e9ed0699865-kube-api-access-j9827\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:22.495607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:22.495579    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"
Oct 30 14:45:22.495607 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:22.495594    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp" event={"ID":"b88f1bbb-88c1-4fff-9987-0e9ed0699865","Type":"ContainerDied","Data":"1e29671f42704b399affe2065173ccb1b39a0009141b084ad9fca489cf3b1d42"}
Oct 30 14:45:22.495877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:22.495654    4530 scope.go:122] "RemoveContainer" containerID="5f839323a66b735eb7356919879813597d633194940c1c0f38df4a061d34a7d6"
Oct 30 14:45:22.497386 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:22.497357    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"
Oct 30 14:45:22.497386 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:22.497362    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6" event={"ID":"7a608b93-87cb-4d9f-9d5a-8414d844b98e","Type":"ContainerDied","Data":"0588155eaaafd985056e4f7c4ff2479aa5223a585aae36aba20937ab041ecfeb"}
Oct 30 14:45:22.507710 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:22.507672    4530 scope.go:122] "RemoveContainer" containerID="2cc82b0b805c758917edfb72f2070cf17145960e13a9dd0fd5a7c543528b3cdd"
Oct 30 14:45:22.561042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:22.561017    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"]
Oct 30 14:45:22.575243 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:22.575219    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d05e21-29889525-x8bc6"]
Oct 30 14:45:22.616594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:22.616561    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"]
Oct 30 14:45:22.621879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:22.621859    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889525-scqxp"]
Oct 30 14:45:23.117221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:23.117189    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" path="/var/lib/kubelet/pods/7a608b93-87cb-4d9f-9d5a-8414d844b98e/volumes"
Oct 30 14:45:23.117683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:23.117667    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" path="/var/lib/kubelet/pods/b88f1bbb-88c1-4fff-9987-0e9ed0699865/volumes"
Oct 30 14:45:26.127170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:26.127133    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"]
Oct 30 14:45:26.127603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:26.127335    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy" containerID="cri-o://0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80" gracePeriod=30
Oct 30 14:45:26.492453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:26.492372    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:45:27.420364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:27.420325    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:45:27.420869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:27.420401    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:45:31.494385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.494353    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:45:31.573260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.573223    4530 generic.go:356] "Generic (PLEG): container finished" podID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerID="0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80" exitCode=0
Oct 30 14:45:31.573474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.573275    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"
Oct 30 14:45:31.573474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.573307    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" event={"ID":"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7","Type":"ContainerDied","Data":"0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80"}
Oct 30 14:45:31.573474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.573362    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v" event={"ID":"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7","Type":"ContainerDied","Data":"be8bb9d06b029bb43d9ba1b6afe816f2a8c63a4a9a3a65f65af3cb0e16e52fc2"}
Oct 30 14:45:31.573474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.573391    4530 scope.go:122] "RemoveContainer" containerID="0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80"
Oct 30 14:45:31.588182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.588157    4530 scope.go:122] "RemoveContainer" containerID="13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d"
Oct 30 14:45:31.600564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.600523    4530 scope.go:122] "RemoveContainer" containerID="0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80"
Oct 30 14:45:31.601085 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:31.601050    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80\": container with ID starting with 0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80 not found: ID does not exist" containerID="0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80"
Oct 30 14:45:31.601205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.601094    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80"} err="failed to get container status \"0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80\": rpc error: code = NotFound desc = could not find container \"0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80\": container with ID starting with 0a91dc3f6a14f8720ed07b560be0a37c165dacc06f98e2be840fce9b0ceb1e80 not found: ID does not exist"
Oct 30 14:45:31.601205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.601120    4530 scope.go:122] "RemoveContainer" containerID="13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d"
Oct 30 14:45:31.601485 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:31.601454    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d\": container with ID starting with 13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d not found: ID does not exist" containerID="13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d"
Oct 30 14:45:31.601647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.601491    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d"} err="failed to get container status \"13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d\": rpc error: code = NotFound desc = could not find container \"13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d\": container with ID starting with 13e4694cc8fbb93705a5b4e6282ff4dab4ea14677328e5ea106e3aec23866f7d not found: ID does not exist"
Oct 30 14:45:31.606091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.606063    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-kube-api-access-992r4\" (UniqueName: \"kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-kube-api-access-992r4\") pod \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") "
Oct 30 14:45:31.606219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.606123    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-token\" (UniqueName: \"kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-token\") pod \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") "
Oct 30 14:45:31.606219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.606201    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-envoy\") pod \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") "
Oct 30 14:45:31.606352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.606231    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-data\") pod \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") "
Oct 30 14:45:31.606412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.606386    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-podinfo\") pod \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") "
Oct 30 14:45:31.606504 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.606488    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-socket\") pod \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") "
Oct 30 14:45:31.606764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.606575    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istiod-ca-cert\") pod \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") "
Oct 30 14:45:31.606764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.606623    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-certs\") pod \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") "
Oct 30 14:45:31.606764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.606621    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-data" pod "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" (UID: "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:31.606764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.606663    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-credential-socket\") pod \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\" (UID: \"cdc1ef01-f2c5-42ae-9e3d-690cbed016f7\") "
Oct 30 14:45:31.607007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.606849    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-certs" pod "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" (UID: "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:31.607007 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.606917    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istiod-ca-cert" pod "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" (UID: "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:45:31.607106 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.607061    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-credential-socket" pod "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" (UID: "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:31.607236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.607209    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-socket" pod "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" (UID: "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:31.607363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.607351    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:31.607432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.607365    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:31.607432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.607378    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:31.607432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.607391    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:31.607432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.607404    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:31.609712 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.609686    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-envoy" pod "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" (UID: "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:45:31.609855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.609837    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-kube-api-access-992r4" pod "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" (UID: "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7"). InnerVolumeSpecName "kube-api-access-992r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:31.610931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.610903    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-token" pod "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" (UID: "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:31.619877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.619847    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-podinfo" pod "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" (UID: "cdc1ef01-f2c5-42ae-9e3d-690cbed016f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:45:31.708683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.708642    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-992r4\" (UniqueName: \"kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-kube-api-access-992r4\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:31.708683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.708675    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:31.708683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.708685    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:31.708683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.708694    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:31.938060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.938026    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"]
Oct 30 14:45:31.952512 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:31.952469    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-nl96v"]
Oct 30 14:45:33.117602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:33.117561    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" path="/var/lib/kubelet/pods/cdc1ef01-f2c5-42ae-9e3d-690cbed016f7/volumes"
Oct 30 14:45:35.389381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:35.389350    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9dw4d_c340cff3-c8aa-40df-869a-5d5bf4c7210b/payload-processing/0.log"
Oct 30 14:45:35.584403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:35.584374    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9dw4d_c340cff3-c8aa-40df-869a-5d5bf4c7210b/payload-processing/0.log"
Oct 30 14:45:41.428759 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:41.428717    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:45:41.877522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.877484    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"]
Oct 30 14:45:41.878249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878231    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" containerName="cleanup"
Oct 30 14:45:41.878295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878251    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" containerName="cleanup"
Oct 30 14:45:41.878295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878263    4530 cpu_manager.go:414] "containerMap: removing container" podUID="24867fc7-ab36-491e-bccf-64a34aa01681" containerName="cleanup"
Oct 30 14:45:41.878295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878268    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24867fc7-ab36-491e-bccf-64a34aa01681" containerName="cleanup"
Oct 30 14:45:41.878295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878278    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy"
Oct 30 14:45:41.878295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878285    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy"
Oct 30 14:45:41.878295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878293    4530 cpu_manager.go:414] "containerMap: removing container" podUID="24867fc7-ab36-491e-bccf-64a34aa01681" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878298    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24867fc7-ab36-491e-bccf-64a34aa01681" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878306    4530 cpu_manager.go:414] "containerMap: removing container" podUID="24867fc7-ab36-491e-bccf-64a34aa01681" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878310    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24867fc7-ab36-491e-bccf-64a34aa01681" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878322    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878328    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878334    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6bd36ed1-3a87-445f-a966-77792741b82c" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878338    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd36ed1-3a87-445f-a966-77792741b82c" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878346    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6bd36ed1-3a87-445f-a966-77792741b82c" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878350    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd36ed1-3a87-445f-a966-77792741b82c" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878359    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878364    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878370    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878374    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878391    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878395    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878403    4530 cpu_manager.go:414] "containerMap: removing container" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878407    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878415    4530 cpu_manager.go:414] "containerMap: removing container" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878419    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878430    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878434    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878446    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" containerName="cleanup"
Oct 30 14:45:41.878500 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878451    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878571    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd36ed1-3a87-445f-a966-77792741b82c" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878581    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878586    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878593    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878600    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878606    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878611    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878615    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878621    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878628    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878635    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878640    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878647    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="24867fc7-ab36-491e-bccf-64a34aa01681" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878653    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878659    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="24867fc7-ab36-491e-bccf-64a34aa01681" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878667    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="24867fc7-ab36-491e-bccf-64a34aa01681" containerName="cleanup"
Oct 30 14:45:41.879203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.878673    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd36ed1-3a87-445f-a966-77792741b82c" containerName="cleanup"
Oct 30 14:45:41.884099 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.884078    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"
Oct 30 14:45:41.902381 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.902348    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"]
Oct 30 14:45:41.910216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.910192    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d05e21-dockercfg-lb2ng\""
Oct 30 14:45:41.938502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.938468    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d05e21\""
Oct 30 14:45:41.970734 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.970700    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs\" (UID: \"f59a9fe9-5a43-4fb8-8446-6b13af4457e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"
Oct 30 14:45:41.970883 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:41.970769    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh9f7\" (UniqueName: \"kubernetes.io/projected/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-kube-api-access-qh9f7\") pod \"payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs\" (UID: \"f59a9fe9-5a43-4fb8-8446-6b13af4457e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"
Oct 30 14:45:42.071972 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:42.071937    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs\" (UID: \"f59a9fe9-5a43-4fb8-8446-6b13af4457e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"
Oct 30 14:45:42.072154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:42.071985    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9f7\" (UniqueName: \"kubernetes.io/projected/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-kube-api-access-qh9f7\") pod \"payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs\" (UID: \"f59a9fe9-5a43-4fb8-8446-6b13af4457e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"
Oct 30 14:45:42.072708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:42.072685    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs\" (UID: \"f59a9fe9-5a43-4fb8-8446-6b13af4457e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"
Oct 30 14:45:42.099042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:42.099003    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9f7\" (UniqueName: \"kubernetes.io/projected/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-kube-api-access-qh9f7\") pod \"payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs\" (UID: \"f59a9fe9-5a43-4fb8-8446-6b13af4457e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"
Oct 30 14:45:42.200242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:42.200144    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"
Oct 30 14:45:42.515396 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:45:42.507287    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59a9fe9_5a43_4fb8_8446_6b13af4457e9.slice/crio-ea2f4f181dd20327651b59e996138af5281f716bd4389a1940880884ece7edc7 WatchSource:0}: Error finding container ea2f4f181dd20327651b59e996138af5281f716bd4389a1940880884ece7edc7: Status 404 returned error can't find the container with id ea2f4f181dd20327651b59e996138af5281f716bd4389a1940880884ece7edc7
Oct 30 14:45:42.515396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:42.510762    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"]
Oct 30 14:45:42.756977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:42.756878    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs" event={"ID":"f59a9fe9-5a43-4fb8-8446-6b13af4457e9","Type":"ContainerStarted","Data":"59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2"}
Oct 30 14:45:42.756977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:42.756943    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"
Oct 30 14:45:42.756977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:42.756964    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs" event={"ID":"f59a9fe9-5a43-4fb8-8446-6b13af4457e9","Type":"ContainerStarted","Data":"ea2f4f181dd20327651b59e996138af5281f716bd4389a1940880884ece7edc7"}
Oct 30 14:45:42.806215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:42.806162    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs" podStartSLOduration=1.8061472539999999 podStartE2EDuration="1.806147254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:42.803286225 +0000 UTC m=+3407.835185165" watchObservedRunningTime="2026-10-30 14:45:42.806147254 +0000 UTC m=+3407.838046225"
Oct 30 14:45:53.767755 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:53.767716    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"
Oct 30 14:45:54.380567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:54.380235    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:45:56.374473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:56.373939    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"]
Oct 30 14:45:56.374473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:56.374222    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs" podUID="697e817c-3d63-4813-8d5b-84ed291e3f3f" containerName="maas-api" containerID="cri-o://56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa" gracePeriod=30
Oct 30 14:45:57.030247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.030202    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:45:57.077964 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.077927    4530 generic.go:356] "Generic (PLEG): container finished" podID="697e817c-3d63-4813-8d5b-84ed291e3f3f" containerID="56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa" exitCode=0
Oct 30 14:45:57.078210 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.078004    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs" event={"ID":"697e817c-3d63-4813-8d5b-84ed291e3f3f","Type":"ContainerDied","Data":"56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa"}
Oct 30 14:45:57.078210 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.078066    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs" event={"ID":"697e817c-3d63-4813-8d5b-84ed291e3f3f","Type":"ContainerDied","Data":"45dbd7a0eab676616c9bfa25d4f6047706bf255ff214e7cfec57081ad28244a7"}
Oct 30 14:45:57.078210 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.078088    4530 scope.go:122] "RemoveContainer" containerID="56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa"
Oct 30 14:45:57.078210 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.078207    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"
Oct 30 14:45:57.133339 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.133269    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/697e817c-3d63-4813-8d5b-84ed291e3f3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/697e817c-3d63-4813-8d5b-84ed291e3f3f-openshift-service-ca\") pod \"697e817c-3d63-4813-8d5b-84ed291e3f3f\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") "
Oct 30 14:45:57.133613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.133592    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/697e817c-3d63-4813-8d5b-84ed291e3f3f-kube-api-access-lrj9x\" (UniqueName: \"kubernetes.io/projected/697e817c-3d63-4813-8d5b-84ed291e3f3f-kube-api-access-lrj9x\") pod \"697e817c-3d63-4813-8d5b-84ed291e3f3f\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") "
Oct 30 14:45:57.133806 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.133790    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-maas-api-tls\") pod \"697e817c-3d63-4813-8d5b-84ed291e3f3f\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") "
Oct 30 14:45:57.134016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.133984    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/697e817c-3d63-4813-8d5b-84ed291e3f3f-openshift-service-ca" pod "697e817c-3d63-4813-8d5b-84ed291e3f3f" (UID: "697e817c-3d63-4813-8d5b-84ed291e3f3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:45:57.135168 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.135144    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"]
Oct 30 14:45:57.136774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.136754    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697e817c-3d63-4813-8d5b-84ed291e3f3f-kube-api-access-lrj9x" pod "697e817c-3d63-4813-8d5b-84ed291e3f3f" (UID: "697e817c-3d63-4813-8d5b-84ed291e3f3f"). InnerVolumeSpecName "kube-api-access-lrj9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:45:57.137685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.137651    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-maas-api-tls" pod "697e817c-3d63-4813-8d5b-84ed291e3f3f" (UID: "697e817c-3d63-4813-8d5b-84ed291e3f3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:45:57.137940 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.137918    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-metrics-certs\") pod \"697e817c-3d63-4813-8d5b-84ed291e3f3f\" (UID: \"697e817c-3d63-4813-8d5b-84ed291e3f3f\") "
Oct 30 14:45:57.138878 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.138653    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/697e817c-3d63-4813-8d5b-84ed291e3f3f-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:57.138878 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.138766    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrj9x\" (UniqueName: \"kubernetes.io/projected/697e817c-3d63-4813-8d5b-84ed291e3f3f-kube-api-access-lrj9x\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:57.138878 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.138782    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:57.142186 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.142162    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1d6347-rpkcj"]
Oct 30 14:45:57.142328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.142307    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-metrics-certs" pod "697e817c-3d63-4813-8d5b-84ed291e3f3f" (UID: "697e817c-3d63-4813-8d5b-84ed291e3f3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:45:57.143620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.143600    4530 scope.go:122] "RemoveContainer" containerID="56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa"
Oct 30 14:45:57.144011 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:45:57.143992    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa\": container with ID starting with 56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa not found: ID does not exist" containerID="56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa"
Oct 30 14:45:57.144086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.144020    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa"} err="failed to get container status \"56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa\": rpc error: code = NotFound desc = could not find container \"56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa\": container with ID starting with 56f3230ad7f7ec461c433e7ef75781a2480eb159f4a86728740ce3a7a0f711fa not found: ID does not exist"
Oct 30 14:45:57.242619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.242361    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/697e817c-3d63-4813-8d5b-84ed291e3f3f-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:45:57.461504 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.461471    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"]
Oct 30 14:45:57.478705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:57.478668    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs"]
Oct 30 14:45:59.120434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:59.120395    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="697e817c-3d63-4813-8d5b-84ed291e3f3f" path="/var/lib/kubelet/pods/697e817c-3d63-4813-8d5b-84ed291e3f3f/volumes"
Oct 30 14:45:59.121177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:59.121152    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf3b7066-08b0-4504-a19f-e66982a945df" path="/var/lib/kubelet/pods/bf3b7066-08b0-4504-a19f-e66982a945df/volumes"
Oct 30 14:45:59.969995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:45:59.969954    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-1d6347-56799647c4-shbrs" podUID="697e817c-3d63-4813-8d5b-84ed291e3f3f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.246:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:46:01.819161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:01.819111    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f_d4ce80eb-7159-47c6-a5b5-cf94871d1e46/payload-processing/0.log"
Oct 30 14:46:02.030461 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:02.030426    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9dw4d_c340cff3-c8aa-40df-869a-5d5bf4c7210b/payload-processing/0.log"
Oct 30 14:46:02.215747 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:02.215662    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9dw4d_c340cff3-c8aa-40df-869a-5d5bf4c7210b/payload-processing/0.log"
Oct 30 14:46:02.402974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:02.402945    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s_c1ab0398-256e-4d59-814a-bde23499d65d/payload-processing/0.log"
Oct 30 14:46:02.618897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:02.618865    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s_c1ab0398-256e-4d59-814a-bde23499d65d/payload-processing/0.log"
Oct 30 14:46:04.472070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.472037    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"]
Oct 30 14:46:04.472603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.472297    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk" podUID="72181d49-18db-4413-9c82-82d995de0b43" containerName="maas-api" containerID="cri-o://1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90" gracePeriod=30
Oct 30 14:46:04.685635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.685596    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"]
Oct 30 14:46:04.685874 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.685834    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd" podUID="9c1a6fcb-ea05-4886-a998-df6cce7f3157" containerName="payload-processing" containerID="cri-o://8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac" gracePeriod=30
Oct 30 14:46:04.713774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.710994    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"]
Oct 30 14:46:04.713774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.711232    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47" podUID="4b9dbd02-5964-42ef-a2e7-c15119066eb6" containerName="payload-pre-processing" containerID="cri-o://a75eac50009385dabdabf28b8a2db62a1df7f932f7fbbc011a405c81596f6624" gracePeriod=30
Oct 30 14:46:04.864063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.862696    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:46:04.970902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.970327    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-maas-api-tls\") pod \"72181d49-18db-4413-9c82-82d995de0b43\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") "
Oct 30 14:46:04.970902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.970482    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72181d49-18db-4413-9c82-82d995de0b43-kube-api-access-tr74f\" (UniqueName: \"kubernetes.io/projected/72181d49-18db-4413-9c82-82d995de0b43-kube-api-access-tr74f\") pod \"72181d49-18db-4413-9c82-82d995de0b43\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") "
Oct 30 14:46:04.972563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.971185    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-metrics-certs\") pod \"72181d49-18db-4413-9c82-82d995de0b43\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") "
Oct 30 14:46:04.972563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.971291    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72181d49-18db-4413-9c82-82d995de0b43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72181d49-18db-4413-9c82-82d995de0b43-openshift-service-ca\") pod \"72181d49-18db-4413-9c82-82d995de0b43\" (UID: \"72181d49-18db-4413-9c82-82d995de0b43\") "
Oct 30 14:46:04.973338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.973288    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72181d49-18db-4413-9c82-82d995de0b43-openshift-service-ca" pod "72181d49-18db-4413-9c82-82d995de0b43" (UID: "72181d49-18db-4413-9c82-82d995de0b43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:46:04.974855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.974808    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-maas-api-tls" pod "72181d49-18db-4413-9c82-82d995de0b43" (UID: "72181d49-18db-4413-9c82-82d995de0b43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:46:04.976681 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.976648    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-metrics-certs" pod "72181d49-18db-4413-9c82-82d995de0b43" (UID: "72181d49-18db-4413-9c82-82d995de0b43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:46:04.978020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:04.977988    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72181d49-18db-4413-9c82-82d995de0b43-kube-api-access-tr74f" pod "72181d49-18db-4413-9c82-82d995de0b43" (UID: "72181d49-18db-4413-9c82-82d995de0b43"). InnerVolumeSpecName "kube-api-access-tr74f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:46:05.042894 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.042827    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"
Oct 30 14:46:05.088206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.082035    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1a6fcb-ea05-4886-a998-df6cce7f3157-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1a6fcb-ea05-4886-a998-df6cce7f3157-plugins-config-volume\") pod \"9c1a6fcb-ea05-4886-a998-df6cce7f3157\" (UID: \"9c1a6fcb-ea05-4886-a998-df6cce7f3157\") "
Oct 30 14:46:05.088206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.082165    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1a6fcb-ea05-4886-a998-df6cce7f3157-kube-api-access-d4mp8\" (UniqueName: \"kubernetes.io/projected/9c1a6fcb-ea05-4886-a998-df6cce7f3157-kube-api-access-d4mp8\") pod \"9c1a6fcb-ea05-4886-a998-df6cce7f3157\" (UID: \"9c1a6fcb-ea05-4886-a998-df6cce7f3157\") "
Oct 30 14:46:05.088206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.082804    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72181d49-18db-4413-9c82-82d995de0b43-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:05.088206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.082831    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:05.088206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.082845    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr74f\" (UniqueName: \"kubernetes.io/projected/72181d49-18db-4413-9c82-82d995de0b43-kube-api-access-tr74f\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:05.088206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.082857    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72181d49-18db-4413-9c82-82d995de0b43-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:05.128575 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.128002    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1a6fcb-ea05-4886-a998-df6cce7f3157-plugins-config-volume" pod "9c1a6fcb-ea05-4886-a998-df6cce7f3157" (UID: "9c1a6fcb-ea05-4886-a998-df6cce7f3157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:46:05.194564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.191646    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1a6fcb-ea05-4886-a998-df6cce7f3157-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:05.270930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.267858    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1a6fcb-ea05-4886-a998-df6cce7f3157-kube-api-access-d4mp8" pod "9c1a6fcb-ea05-4886-a998-df6cce7f3157" (UID: "9c1a6fcb-ea05-4886-a998-df6cce7f3157"). InnerVolumeSpecName "kube-api-access-d4mp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:46:05.275147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.273515    4530 generic.go:356] "Generic (PLEG): container finished" podID="9c1a6fcb-ea05-4886-a998-df6cce7f3157" containerID="8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac" exitCode=0
Oct 30 14:46:05.275147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.273658    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd" event={"ID":"9c1a6fcb-ea05-4886-a998-df6cce7f3157","Type":"ContainerDied","Data":"8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac"}
Oct 30 14:46:05.275147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.273712    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd" event={"ID":"9c1a6fcb-ea05-4886-a998-df6cce7f3157","Type":"ContainerDied","Data":"e5c4ac1dce862a18e0e36ac1ead1f2e85e0f67e999cbf9827ae818b20c166c7c"}
Oct 30 14:46:05.275147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.273741    4530 scope.go:122] "RemoveContainer" containerID="8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac"
Oct 30 14:46:05.275147 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.273912    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"
Oct 30 14:46:05.282592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.281986    4530 generic.go:356] "Generic (PLEG): container finished" podID="4b9dbd02-5964-42ef-a2e7-c15119066eb6" containerID="a75eac50009385dabdabf28b8a2db62a1df7f932f7fbbc011a405c81596f6624" exitCode=0
Oct 30 14:46:05.282592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.282163    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47" event={"ID":"4b9dbd02-5964-42ef-a2e7-c15119066eb6","Type":"ContainerDied","Data":"a75eac50009385dabdabf28b8a2db62a1df7f932f7fbbc011a405c81596f6624"}
Oct 30 14:46:05.301342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.299916    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4mp8\" (UniqueName: \"kubernetes.io/projected/9c1a6fcb-ea05-4886-a998-df6cce7f3157-kube-api-access-d4mp8\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:05.311740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.310296    4530 generic.go:356] "Generic (PLEG): container finished" podID="72181d49-18db-4413-9c82-82d995de0b43" containerID="1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90" exitCode=0
Oct 30 14:46:05.311740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.310353    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk" event={"ID":"72181d49-18db-4413-9c82-82d995de0b43","Type":"ContainerDied","Data":"1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90"}
Oct 30 14:46:05.311740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.310400    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk" event={"ID":"72181d49-18db-4413-9c82-82d995de0b43","Type":"ContainerDied","Data":"bb9264a33fa4f5fc050c002a86ee0c9a47ec813cacec5d151f1145a807c519b1"}
Oct 30 14:46:05.311740 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.310474    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"
Oct 30 14:46:05.316303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.316172    4530 scope.go:122] "RemoveContainer" containerID="8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac"
Oct 30 14:46:05.316882 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:46:05.316722    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac\": container with ID starting with 8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac not found: ID does not exist" containerID="8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac"
Oct 30 14:46:05.316882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.316761    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac"} err="failed to get container status \"8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac\": rpc error: code = NotFound desc = could not find container \"8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac\": container with ID starting with 8994ac9f1f640c30322f0a2c41246c92156a70b9023db54e6327d556afd0aeac not found: ID does not exist"
Oct 30 14:46:05.316882 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.316787    4530 scope.go:122] "RemoveContainer" containerID="1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90"
Oct 30 14:46:05.321496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.321441    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"
Oct 30 14:46:05.332578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.332476    4530 scope.go:122] "RemoveContainer" containerID="1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90"
Oct 30 14:46:05.332941 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:46:05.332900    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90\": container with ID starting with 1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90 not found: ID does not exist" containerID="1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90"
Oct 30 14:46:05.333018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.332933    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90"} err="failed to get container status \"1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90\": rpc error: code = NotFound desc = could not find container \"1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90\": container with ID starting with 1a53009db0fef82bbd397957190cde7d56a83abfd3910c82d1792d21accaaa90 not found: ID does not exist"
Oct 30 14:46:05.385762 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.385696    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"]
Oct 30 14:46:05.396944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.396902    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1854b-fd94f4c4-vhlmk"]
Oct 30 14:46:05.436148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.436102    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"]
Oct 30 14:46:05.441040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.441006    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1854b-58595bff5b-vvvvd"]
Oct 30 14:46:05.503559 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.503486    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9dbd02-5964-42ef-a2e7-c15119066eb6-kube-api-access-jzmhk\" (UniqueName: \"kubernetes.io/projected/4b9dbd02-5964-42ef-a2e7-c15119066eb6-kube-api-access-jzmhk\") pod \"4b9dbd02-5964-42ef-a2e7-c15119066eb6\" (UID: \"4b9dbd02-5964-42ef-a2e7-c15119066eb6\") "
Oct 30 14:46:05.504071 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.503996    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9dbd02-5964-42ef-a2e7-c15119066eb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9dbd02-5964-42ef-a2e7-c15119066eb6-plugins-config-volume\") pod \"4b9dbd02-5964-42ef-a2e7-c15119066eb6\" (UID: \"4b9dbd02-5964-42ef-a2e7-c15119066eb6\") "
Oct 30 14:46:05.505166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.505137    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9dbd02-5964-42ef-a2e7-c15119066eb6-plugins-config-volume" pod "4b9dbd02-5964-42ef-a2e7-c15119066eb6" (UID: "4b9dbd02-5964-42ef-a2e7-c15119066eb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:46:05.512096 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.508775    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9dbd02-5964-42ef-a2e7-c15119066eb6-kube-api-access-jzmhk" pod "4b9dbd02-5964-42ef-a2e7-c15119066eb6" (UID: "4b9dbd02-5964-42ef-a2e7-c15119066eb6"). InnerVolumeSpecName "kube-api-access-jzmhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:46:05.611042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.610947    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzmhk\" (UniqueName: \"kubernetes.io/projected/4b9dbd02-5964-42ef-a2e7-c15119066eb6-kube-api-access-jzmhk\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:05.611369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:05.611311    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9dbd02-5964-42ef-a2e7-c15119066eb6-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:06.322961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:06.322927    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"
Oct 30 14:46:06.322961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:06.322944    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47" event={"ID":"4b9dbd02-5964-42ef-a2e7-c15119066eb6","Type":"ContainerDied","Data":"bf18239b2c1e874fd44d3a3cfc3185d72d30571a2d769801709cfb71dc6b4b67"}
Oct 30 14:46:06.323212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:06.323001    4530 scope.go:122] "RemoveContainer" containerID="a75eac50009385dabdabf28b8a2db62a1df7f932f7fbbc011a405c81596f6624"
Oct 30 14:46:06.388110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:06.388074    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"]
Oct 30 14:46:06.392729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:06.392700    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-64bc599575-prc47"]
Oct 30 14:46:07.116637 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:07.116608    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9dbd02-5964-42ef-a2e7-c15119066eb6" path="/var/lib/kubelet/pods/4b9dbd02-5964-42ef-a2e7-c15119066eb6/volumes"
Oct 30 14:46:07.117180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:07.117164    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72181d49-18db-4413-9c82-82d995de0b43" path="/var/lib/kubelet/pods/72181d49-18db-4413-9c82-82d995de0b43/volumes"
Oct 30 14:46:07.117620 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:07.117604    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1a6fcb-ea05-4886-a998-df6cce7f3157" path="/var/lib/kubelet/pods/9c1a6fcb-ea05-4886-a998-df6cce7f3157/volumes"
Oct 30 14:46:07.334166 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:07.334133    4530 generic.go:356] "Generic (PLEG): container finished" podID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerID="20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1" exitCode=0
Oct 30 14:46:07.334342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:07.334195    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" event={"ID":"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f","Type":"ContainerDied","Data":"20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1"}
Oct 30 14:46:07.334701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:07.334683    4530 scope.go:122] "RemoveContainer" containerID="20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1"
Oct 30 14:46:07.336787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:07.336757    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:46:07.336898 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:07.336824    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:46:07.336898 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:07.336883    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:46:08.352215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:08.352179    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" event={"ID":"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f","Type":"ContainerStarted","Data":"56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf"}
Oct 30 14:46:09.013949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:09.013902    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:46:09.014173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:09.014149    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 14:46:09.014221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:09.014200    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 14:46:10.017563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:10.014507    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 14:46:10.017563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:10.014591    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 14:46:11.014583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:11.014515    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 14:46:11.014761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:11.014615    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 14:46:11.369417 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:46:11.369380    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:46:12.014697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:12.014653    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 14:46:12.014914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:12.014714    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 14:46:13.014754 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:13.014716    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 14:46:13.015284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:13.014773    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 14:46:14.014704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:14.014662    4530 patch_prober.go:29] interesting pod/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 14:46:14.014890 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:14.014719    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 14:46:15.019516 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:15.019478    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:46:15.019995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:15.019893    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:46:15.020506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:15.020489    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:46:16.715564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:16.715196    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"]
Oct 30 14:46:21.716434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:21.716361    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj" podUID="9c668c61-9ca2-4958-a9cc-e1d274b422a7" containerName="payload-processing" containerID="cri-o://5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d" gracePeriod=25
Oct 30 14:46:27.465089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:27.465045    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj" podUID="9c668c61-9ca2-4958-a9cc-e1d274b422a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:46:36.097236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:36.097201    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"]
Oct 30 14:46:36.098399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:36.098362    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy" containerID="cri-o://16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05" gracePeriod=30
Oct 30 14:46:37.464764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:37.464724    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj" podUID="9c668c61-9ca2-4958-a9cc-e1d274b422a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:46:41.372142 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:46:41.372055    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:46:41.384219 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.384194    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:46:41.440355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440330    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-credential-socket\") pod \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") "
Oct 30 14:46:41.440566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440366    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-envoy\") pod \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") "
Oct 30 14:46:41.440566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440389    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-certs\") pod \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") "
Oct 30 14:46:41.440566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440433    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-kube-api-access-qb4qt\" (UniqueName: \"kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-kube-api-access-qb4qt\") pod \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") "
Oct 30 14:46:41.440566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440513    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istiod-ca-cert\") pod \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") "
Oct 30 14:46:41.440831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440579    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-data\") pod \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") "
Oct 30 14:46:41.440831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440638    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-credential-socket" pod "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" (UID: "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:46:41.440831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440665    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-socket\") pod \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") "
Oct 30 14:46:41.440831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440700    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-token\" (UniqueName: \"kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-token\") pod \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") "
Oct 30 14:46:41.440831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440741    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-podinfo\") pod \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\" (UID: \"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4\") "
Oct 30 14:46:41.441124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440854    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-certs" pod "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" (UID: "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:46:41.441124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440882    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-data" pod "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" (UID: "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:46:41.441124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.440967    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istiod-ca-cert" pod "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" (UID: "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:46:41.441293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.441144    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-socket" pod "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" (UID: "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:46:41.441565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.441521    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:41.441674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.441576    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:41.441674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.441591    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:41.441674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.441604    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:41.441674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.441617    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:41.442947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.442916    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-envoy" pod "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" (UID: "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:46:41.443086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.442984    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-podinfo" pod "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" (UID: "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:46:41.443086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.442994    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-token" pod "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" (UID: "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:46:41.443086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.443016    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-kube-api-access-qb4qt" pod "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" (UID: "17140e84-9ba9-4ecc-83cf-9498e2ccd8b4"). InnerVolumeSpecName "kube-api-access-qb4qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:46:41.543145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.543110    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:41.543145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.543140    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb4qt\" (UniqueName: \"kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-kube-api-access-qb4qt\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:41.543145 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.543150    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:41.543380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.543160    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:41.797598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.797562    4530 generic.go:356] "Generic (PLEG): container finished" podID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerID="16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05" exitCode=0
Oct 30 14:46:41.797830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.797623    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"
Oct 30 14:46:41.797830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.797642    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" event={"ID":"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4","Type":"ContainerDied","Data":"16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05"}
Oct 30 14:46:41.797830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.797701    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9" event={"ID":"17140e84-9ba9-4ecc-83cf-9498e2ccd8b4","Type":"ContainerDied","Data":"6758f07759eaf05980aaa55e1f90460b4d8331fe395ecd8c28e35070ef44c173"}
Oct 30 14:46:41.797830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.797731    4530 scope.go:122] "RemoveContainer" containerID="16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05"
Oct 30 14:46:41.831116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.831094    4530 scope.go:122] "RemoveContainer" containerID="2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9"
Oct 30 14:46:41.844532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.844507    4530 scope.go:122] "RemoveContainer" containerID="16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05"
Oct 30 14:46:41.844882 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:46:41.844853    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05\": container with ID starting with 16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05 not found: ID does not exist" containerID="16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05"
Oct 30 14:46:41.844966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.844893    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05"} err="failed to get container status \"16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05\": rpc error: code = NotFound desc = could not find container \"16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05\": container with ID starting with 16d9f6160e0f97ab0dda5ce06b62b828c61c2f2656531c5a02e655f0a170cf05 not found: ID does not exist"
Oct 30 14:46:41.844966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.844922    4530 scope.go:122] "RemoveContainer" containerID="2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9"
Oct 30 14:46:41.845280 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:46:41.845248    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9\": container with ID starting with 2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9 not found: ID does not exist" containerID="2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9"
Oct 30 14:46:41.845351 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.845287    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9"} err="failed to get container status \"2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9\": rpc error: code = NotFound desc = could not find container \"2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9\": container with ID starting with 2095ad3af0fc0a22c0bfcee262645b6d31236e3b4b3a892bc008c0c0ac060ff9 not found: ID does not exist"
Oct 30 14:46:41.865813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.865768    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"]
Oct 30 14:46:41.873073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.873043    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4be240-openshift-default-5b576754f5-9nph9"]
Oct 30 14:46:41.977661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:41.977634    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"
Oct 30 14:46:42.052307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.052278    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c668c61-9ca2-4958-a9cc-e1d274b422a7-kube-api-access-7dtk5\" (UniqueName: \"kubernetes.io/projected/9c668c61-9ca2-4958-a9cc-e1d274b422a7-kube-api-access-7dtk5\") pod \"9c668c61-9ca2-4958-a9cc-e1d274b422a7\" (UID: \"9c668c61-9ca2-4958-a9cc-e1d274b422a7\") "
Oct 30 14:46:42.052487 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.052340    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c668c61-9ca2-4958-a9cc-e1d274b422a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c668c61-9ca2-4958-a9cc-e1d274b422a7-plugins-config-volume\") pod \"9c668c61-9ca2-4958-a9cc-e1d274b422a7\" (UID: \"9c668c61-9ca2-4958-a9cc-e1d274b422a7\") "
Oct 30 14:46:42.052867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.052834    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c668c61-9ca2-4958-a9cc-e1d274b422a7-plugins-config-volume" pod "9c668c61-9ca2-4958-a9cc-e1d274b422a7" (UID: "9c668c61-9ca2-4958-a9cc-e1d274b422a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:46:42.053112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.053090    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c668c61-9ca2-4958-a9cc-e1d274b422a7-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:42.054657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.054631    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c668c61-9ca2-4958-a9cc-e1d274b422a7-kube-api-access-7dtk5" pod "9c668c61-9ca2-4958-a9cc-e1d274b422a7" (UID: "9c668c61-9ca2-4958-a9cc-e1d274b422a7"). InnerVolumeSpecName "kube-api-access-7dtk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:46:42.154090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.154054    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dtk5\" (UniqueName: \"kubernetes.io/projected/9c668c61-9ca2-4958-a9cc-e1d274b422a7-kube-api-access-7dtk5\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:42.806743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.806707    4530 generic.go:356] "Generic (PLEG): container finished" podID="9c668c61-9ca2-4958-a9cc-e1d274b422a7" containerID="5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d" exitCode=0
Oct 30 14:46:42.807204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.806751    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"
Oct 30 14:46:42.807204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.806811    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj" event={"ID":"9c668c61-9ca2-4958-a9cc-e1d274b422a7","Type":"ContainerDied","Data":"5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d"}
Oct 30 14:46:42.807204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.806881    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj" event={"ID":"9c668c61-9ca2-4958-a9cc-e1d274b422a7","Type":"ContainerDied","Data":"e7e5b5c12d0d79d53320d52dd60f32a8337263658af69220081d7264f0e8e8c1"}
Oct 30 14:46:42.807204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.806906    4530 scope.go:122] "RemoveContainer" containerID="5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d"
Oct 30 14:46:42.817937 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.817920    4530 scope.go:122] "RemoveContainer" containerID="5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d"
Oct 30 14:46:42.818214 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:46:42.818192    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d\": container with ID starting with 5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d not found: ID does not exist" containerID="5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d"
Oct 30 14:46:42.818299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.818224    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d"} err="failed to get container status \"5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d\": rpc error: code = NotFound desc = could not find container \"5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d\": container with ID starting with 5102b65287ac9b61cdeac2bf41c4869df31431681ab0997951bae3fbcbec2c7d not found: ID does not exist"
Oct 30 14:46:42.870686 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.870655    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"]
Oct 30 14:46:42.875246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:42.875222    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a7bb0c-59c8dfbc9d-nvhdj"]
Oct 30 14:46:43.116311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:43.116231    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" path="/var/lib/kubelet/pods/17140e84-9ba9-4ecc-83cf-9498e2ccd8b4/volumes"
Oct 30 14:46:43.116743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:43.116728    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c668c61-9ca2-4958-a9cc-e1d274b422a7" path="/var/lib/kubelet/pods/9c668c61-9ca2-4958-a9cc-e1d274b422a7/volumes"
Oct 30 14:46:47.042696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.042656    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"]
Oct 30 14:46:47.043171 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.042900    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" containerName="maas-api" containerID="cri-o://dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee" gracePeriod=30
Oct 30 14:46:47.336134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.336106    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:46:47.452989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.452952    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c16cc6ed-d030-440f-97dc-49d67415754b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16cc6ed-d030-440f-97dc-49d67415754b-openshift-service-ca\") pod \"c16cc6ed-d030-440f-97dc-49d67415754b\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") "
Oct 30 14:46:47.453164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.453012    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-maas-api-tls\") pod \"c16cc6ed-d030-440f-97dc-49d67415754b\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") "
Oct 30 14:46:47.453164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.453125    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16cc6ed-d030-440f-97dc-49d67415754b-kube-api-access-tsw7z\" (UniqueName: \"kubernetes.io/projected/c16cc6ed-d030-440f-97dc-49d67415754b-kube-api-access-tsw7z\") pod \"c16cc6ed-d030-440f-97dc-49d67415754b\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") "
Oct 30 14:46:47.453273 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.453161    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-metrics-certs\") pod \"c16cc6ed-d030-440f-97dc-49d67415754b\" (UID: \"c16cc6ed-d030-440f-97dc-49d67415754b\") "
Oct 30 14:46:47.453357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.453332    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16cc6ed-d030-440f-97dc-49d67415754b-openshift-service-ca" pod "c16cc6ed-d030-440f-97dc-49d67415754b" (UID: "c16cc6ed-d030-440f-97dc-49d67415754b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:46:47.453858 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.453817    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16cc6ed-d030-440f-97dc-49d67415754b-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:47.455299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.455271    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-maas-api-tls" pod "c16cc6ed-d030-440f-97dc-49d67415754b" (UID: "c16cc6ed-d030-440f-97dc-49d67415754b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:46:47.455299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.455290    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-metrics-certs" pod "c16cc6ed-d030-440f-97dc-49d67415754b" (UID: "c16cc6ed-d030-440f-97dc-49d67415754b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:46:47.455418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.455314    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16cc6ed-d030-440f-97dc-49d67415754b-kube-api-access-tsw7z" pod "c16cc6ed-d030-440f-97dc-49d67415754b" (UID: "c16cc6ed-d030-440f-97dc-49d67415754b"). InnerVolumeSpecName "kube-api-access-tsw7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:46:47.556109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.556010    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:47.556109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.556052    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsw7z\" (UniqueName: \"kubernetes.io/projected/c16cc6ed-d030-440f-97dc-49d67415754b-kube-api-access-tsw7z\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:47.556109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.556069    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16cc6ed-d030-440f-97dc-49d67415754b-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:47.879345 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.879241    4530 generic.go:356] "Generic (PLEG): container finished" podID="c16cc6ed-d030-440f-97dc-49d67415754b" containerID="dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee" exitCode=0
Oct 30 14:46:47.879529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.879424    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" event={"ID":"c16cc6ed-d030-440f-97dc-49d67415754b","Type":"ContainerDied","Data":"dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee"}
Oct 30 14:46:47.879529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.879488    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw" event={"ID":"c16cc6ed-d030-440f-97dc-49d67415754b","Type":"ContainerDied","Data":"8326013577aed472de9d4704bdfcf7b1428dd6be20a381846534a13e8b4d8f9b"}
Oct 30 14:46:47.879529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.879522    4530 scope.go:122] "RemoveContainer" containerID="dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee"
Oct 30 14:46:47.879863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.879834    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"
Oct 30 14:46:47.893654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.893632    4530 scope.go:122] "RemoveContainer" containerID="d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c"
Oct 30 14:46:47.903761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.903743    4530 scope.go:122] "RemoveContainer" containerID="dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee"
Oct 30 14:46:47.904052 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:46:47.904032    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee\": container with ID starting with dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee not found: ID does not exist" containerID="dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee"
Oct 30 14:46:47.904109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.904060    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee"} err="failed to get container status \"dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee\": rpc error: code = NotFound desc = could not find container \"dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee\": container with ID starting with dc195cbbefcaa128147800855c24bb5dfe97fca9fec660fabda5b28491639fee not found: ID does not exist"
Oct 30 14:46:47.904109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.904077    4530 scope.go:122] "RemoveContainer" containerID="d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c"
Oct 30 14:46:47.904360 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:46:47.904341    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c\": container with ID starting with d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c not found: ID does not exist" containerID="d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c"
Oct 30 14:46:47.904407 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.904367    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c"} err="failed to get container status \"d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c\": rpc error: code = NotFound desc = could not find container \"d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c\": container with ID starting with d35fc532fc383255af48cd353010db310f08a48ec75f69250e5a473ff712623c not found: ID does not exist"
Oct 30 14:46:47.945190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.945160    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"]
Oct 30 14:46:47.959515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:47.959483    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-396d70-786b76f946-6bftw"]
Oct 30 14:46:49.115729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:49.115695    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" path="/var/lib/kubelet/pods/c16cc6ed-d030-440f-97dc-49d67415754b/volumes"
Oct 30 14:46:50.883705 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:50.883668    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"]
Oct 30 14:46:50.884238 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:50.883878    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7" podUID="18d70ff9-ac45-4052-a9a2-e9a6bd0af192" containerName="payload-processing" containerID="cri-o://6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0" gracePeriod=30
Oct 30 14:46:51.204684 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.204652    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"
Oct 30 14:46:51.221180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.221154    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-plugins-config-volume\") pod \"18d70ff9-ac45-4052-a9a2-e9a6bd0af192\" (UID: \"18d70ff9-ac45-4052-a9a2-e9a6bd0af192\") "
Oct 30 14:46:51.221302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.221194    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-kube-api-access-twlsq\" (UniqueName: \"kubernetes.io/projected/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-kube-api-access-twlsq\") pod \"18d70ff9-ac45-4052-a9a2-e9a6bd0af192\" (UID: \"18d70ff9-ac45-4052-a9a2-e9a6bd0af192\") "
Oct 30 14:46:51.221581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.221525    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-plugins-config-volume" pod "18d70ff9-ac45-4052-a9a2-e9a6bd0af192" (UID: "18d70ff9-ac45-4052-a9a2-e9a6bd0af192"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:46:51.222083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.222056    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:51.223291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.223265    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-kube-api-access-twlsq" pod "18d70ff9-ac45-4052-a9a2-e9a6bd0af192" (UID: "18d70ff9-ac45-4052-a9a2-e9a6bd0af192"). InnerVolumeSpecName "kube-api-access-twlsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:46:51.332601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.332558    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twlsq\" (UniqueName: \"kubernetes.io/projected/18d70ff9-ac45-4052-a9a2-e9a6bd0af192-kube-api-access-twlsq\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:46:51.967331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.967300    4530 generic.go:356] "Generic (PLEG): container finished" podID="18d70ff9-ac45-4052-a9a2-e9a6bd0af192" containerID="6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0" exitCode=0
Oct 30 14:46:51.967784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.967340    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"
Oct 30 14:46:51.967784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.967381    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7" event={"ID":"18d70ff9-ac45-4052-a9a2-e9a6bd0af192","Type":"ContainerDied","Data":"6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0"}
Oct 30 14:46:51.967784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.967435    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7" event={"ID":"18d70ff9-ac45-4052-a9a2-e9a6bd0af192","Type":"ContainerDied","Data":"9e6739736098569b9e9e38439a4bf1c887fb42dffc92ac54d97b77cfbc32bfb3"}
Oct 30 14:46:51.967784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.967458    4530 scope.go:122] "RemoveContainer" containerID="6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0"
Oct 30 14:46:51.978303 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.978283    4530 scope.go:122] "RemoveContainer" containerID="6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0"
Oct 30 14:46:51.978638 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:46:51.978523    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0\": container with ID starting with 6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0 not found: ID does not exist" containerID="6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0"
Oct 30 14:46:51.978638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:51.978564    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0"} err="failed to get container status \"6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0\": rpc error: code = NotFound desc = could not find container \"6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0\": container with ID starting with 6fe3bd92564a9d953ade13c602e4687627d9f082681aaf9860b959c6cfe73db0 not found: ID does not exist"
Oct 30 14:46:52.057012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:52.056978    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"]
Oct 30 14:46:52.068326 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:52.068296    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c209-59b7c984c4-9wlm7"]
Oct 30 14:46:53.115883 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:46:53.115848    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d70ff9-ac45-4052-a9a2-e9a6bd0af192" path="/var/lib/kubelet/pods/18d70ff9-ac45-4052-a9a2-e9a6bd0af192/volumes"
Oct 30 14:47:04.978996 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:04.978955    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:47:05.216753 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.216717    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"]
Oct 30 14:47:05.218031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.218001    4530 cpu_manager.go:414] "containerMap: removing container" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450" containerName="cleanup"
Oct 30 14:47:05.218212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.218197    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0686b700-1a3d-4a49-82e0-73d5ac9de450" containerName="cleanup"
Oct 30 14:47:05.218317 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.218306    4530 cpu_manager.go:414] "containerMap: removing container" podUID="72181d49-18db-4413-9c82-82d995de0b43" containerName="maas-api"
Oct 30 14:47:05.218400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.218388    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72181d49-18db-4413-9c82-82d995de0b43" containerName="maas-api"
Oct 30 14:47:05.218499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.218489    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" containerName="cleanup"
Oct 30 14:47:05.218594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.218582    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88f1bbb-88c1-4fff-9987-0e9ed0699865" containerName="cleanup"
Oct 30 14:47:05.218706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.218695    4530 cpu_manager.go:414] "containerMap: removing container" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy"
Oct 30 14:47:05.218810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.218798    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy"
Oct 30 14:47:05.218899 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.218888    4530 cpu_manager.go:414] "containerMap: removing container" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy"
Oct 30 14:47:05.218978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.218968    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy"
Oct 30 14:47:05.219069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.219058    4530 cpu_manager.go:414] "containerMap: removing container" podUID="18d70ff9-ac45-4052-a9a2-e9a6bd0af192" containerName="payload-processing"
Oct 30 14:47:05.219162 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.219151    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d70ff9-ac45-4052-a9a2-e9a6bd0af192" containerName="payload-processing"
Oct 30 14:47:05.219261 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.219251    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9c1a6fcb-ea05-4886-a998-df6cce7f3157" containerName="payload-processing"
Oct 30 14:47:05.219343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.219333    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1a6fcb-ea05-4886-a998-df6cce7f3157" containerName="payload-processing"
Oct 30 14:47:05.219436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.219425    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9c668c61-9ca2-4958-a9cc-e1d274b422a7" containerName="payload-processing"
Oct 30 14:47:05.219514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.219502    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c668c61-9ca2-4958-a9cc-e1d274b422a7" containerName="payload-processing"
Oct 30 14:47:05.219736 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.219721    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" containerName="maas-api"
Oct 30 14:47:05.219821 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.219811    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" containerName="maas-api"
Oct 30 14:47:05.219889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.219880    4530 cpu_manager.go:414] "containerMap: removing container" podUID="4b9dbd02-5964-42ef-a2e7-c15119066eb6" containerName="payload-pre-processing"
Oct 30 14:47:05.219953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.219944    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9dbd02-5964-42ef-a2e7-c15119066eb6" containerName="payload-pre-processing"
Oct 30 14:47:05.220021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.220013    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy"
Oct 30 14:47:05.220088 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.220079    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc1ef01-f2c5-42ae-9e3d-690cbed016f7" containerName="istio-proxy"
Oct 30 14:47:05.220164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.220155    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" containerName="maas-api"
Oct 30 14:47:05.220223 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.220215    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" containerName="maas-api"
Oct 30 14:47:05.220306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.220298    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" containerName="cleanup"
Oct 30 14:47:05.220368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.220359    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" containerName="cleanup"
Oct 30 14:47:05.220437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.220429    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6bd36ed1-3a87-445f-a966-77792741b82c" containerName="cleanup"
Oct 30 14:47:05.220499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.220491    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd36ed1-3a87-445f-a966-77792741b82c" containerName="cleanup"
Oct 30 14:47:05.220606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.220596    4530 cpu_manager.go:414] "containerMap: removing container" podUID="697e817c-3d63-4813-8d5b-84ed291e3f3f" containerName="maas-api"
Oct 30 14:47:05.220677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.220668    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="697e817c-3d63-4813-8d5b-84ed291e3f3f" containerName="maas-api"
Oct 30 14:47:05.220746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.220738    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" containerName="maas-api"
Oct 30 14:47:05.220805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.220796    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" containerName="maas-api"
Oct 30 14:47:05.221036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.221023    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d70ff9-ac45-4052-a9a2-e9a6bd0af192" containerName="payload-processing"
Oct 30 14:47:05.221126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.221118    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1a6fcb-ea05-4886-a998-df6cce7f3157" containerName="payload-processing"
Oct 30 14:47:05.221198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.221190    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c668c61-9ca2-4958-a9cc-e1d274b422a7" containerName="payload-processing"
Oct 30 14:47:05.221267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.221259    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" containerName="maas-api"
Oct 30 14:47:05.221332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.221323    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9dbd02-5964-42ef-a2e7-c15119066eb6" containerName="payload-pre-processing"
Oct 30 14:47:05.221426 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.221417    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd36ed1-3a87-445f-a966-77792741b82c" containerName="cleanup"
Oct 30 14:47:05.221501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.221493    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a608b93-87cb-4d9f-9d5a-8414d844b98e" containerName="cleanup"
Oct 30 14:47:05.221588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.221577    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="697e817c-3d63-4813-8d5b-84ed291e3f3f" containerName="maas-api"
Oct 30 14:47:05.221662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.221651    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" containerName="maas-api"
Oct 30 14:47:05.221752 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.221742    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec96aca-bd5f-4ccc-97e9-d4d63f7bc0b5" containerName="cleanup"
Oct 30 14:47:05.221850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.221838    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="72181d49-18db-4413-9c82-82d995de0b43" containerName="maas-api"
Oct 30 14:47:05.222030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.222010    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy"
Oct 30 14:47:05.222113 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.222103    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="17140e84-9ba9-4ecc-83cf-9498e2ccd8b4" containerName="istio-proxy"
Oct 30 14:47:05.231631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.231565    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"]
Oct 30 14:47:05.232677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.232647    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"
Oct 30 14:47:05.263037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.263000    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bcc710\""
Oct 30 14:47:05.264030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.264003    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bcc710-dockercfg-w9zp8\""
Oct 30 14:47:05.338281 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.338247    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9223ac67-5eec-4e34-a626-d585e44e5444-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq\" (UID: \"9223ac67-5eec-4e34-a626-d585e44e5444\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"
Oct 30 14:47:05.338462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.338423    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fwzk\" (UniqueName: \"kubernetes.io/projected/9223ac67-5eec-4e34-a626-d585e44e5444-kube-api-access-5fwzk\") pod \"payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq\" (UID: \"9223ac67-5eec-4e34-a626-d585e44e5444\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"
Oct 30 14:47:05.407508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.407469    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"]
Oct 30 14:47:05.408364 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.408344    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16cc6ed-d030-440f-97dc-49d67415754b" containerName="maas-api"
Oct 30 14:47:05.412069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.412053    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"
Oct 30 14:47:05.438640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.438592    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"]
Oct 30 14:47:05.439714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.439683    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fwzk\" (UniqueName: \"kubernetes.io/projected/9223ac67-5eec-4e34-a626-d585e44e5444-kube-api-access-5fwzk\") pod \"payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq\" (UID: \"9223ac67-5eec-4e34-a626-d585e44e5444\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"
Oct 30 14:47:05.439827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.439782    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n\" (UID: \"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"
Oct 30 14:47:05.440070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.440048    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9223ac67-5eec-4e34-a626-d585e44e5444-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq\" (UID: \"9223ac67-5eec-4e34-a626-d585e44e5444\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"
Oct 30 14:47:05.440156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.440095    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5n9\" (UniqueName: \"kubernetes.io/projected/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-kube-api-access-cq5n9\") pod \"payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n\" (UID: \"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"
Oct 30 14:47:05.440802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.440781    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9223ac67-5eec-4e34-a626-d585e44e5444-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq\" (UID: \"9223ac67-5eec-4e34-a626-d585e44e5444\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"
Oct 30 14:47:05.490218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.490143    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fwzk\" (UniqueName: \"kubernetes.io/projected/9223ac67-5eec-4e34-a626-d585e44e5444-kube-api-access-5fwzk\") pod \"payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq\" (UID: \"9223ac67-5eec-4e34-a626-d585e44e5444\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"
Oct 30 14:47:05.541047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.541016    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5n9\" (UniqueName: \"kubernetes.io/projected/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-kube-api-access-cq5n9\") pod \"payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n\" (UID: \"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"
Oct 30 14:47:05.541212 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.541124    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n\" (UID: \"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"
Oct 30 14:47:05.541950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.541929    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n\" (UID: \"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"
Oct 30 14:47:05.569279 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.569253    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5n9\" (UniqueName: \"kubernetes.io/projected/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-kube-api-access-cq5n9\") pod \"payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n\" (UID: \"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"
Oct 30 14:47:05.593337 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.593291    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"
Oct 30 14:47:05.726394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.726343    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"
Oct 30 14:47:05.914259 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:47:05.914229    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9223ac67_5eec_4e34_a626_d585e44e5444.slice/crio-4ef3938c532c01b8933f1cf7da8c89d423a8bf3bb81e2be833674da1ae827b05 WatchSource:0}: Error finding container 4ef3938c532c01b8933f1cf7da8c89d423a8bf3bb81e2be833674da1ae827b05: Status 404 returned error can't find the container with id 4ef3938c532c01b8933f1cf7da8c89d423a8bf3bb81e2be833674da1ae827b05
Oct 30 14:47:05.915820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:05.915793    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"]
Oct 30 14:47:06.057253 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:47:06.057220    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda112e79e_e7e5_42c6_a74a_6c63e15fdc1e.slice/crio-1a90a4cd37dc6fd71e2255c7a92a0ebde9841cf8f3e4e048e5c2d93ff9ee28d3 WatchSource:0}: Error finding container 1a90a4cd37dc6fd71e2255c7a92a0ebde9841cf8f3e4e048e5c2d93ff9ee28d3: Status 404 returned error can't find the container with id 1a90a4cd37dc6fd71e2255c7a92a0ebde9841cf8f3e4e048e5c2d93ff9ee28d3
Oct 30 14:47:06.058879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:06.058851    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"]
Oct 30 14:47:06.081508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:06.081473    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq" event={"ID":"9223ac67-5eec-4e34-a626-d585e44e5444","Type":"ContainerStarted","Data":"6d18dd94dba7b6f05e4d45b8f7ed6f07cbe2646847e28f79d165e1b92878437e"}
Oct 30 14:47:06.081673 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:06.081534    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq" event={"ID":"9223ac67-5eec-4e34-a626-d585e44e5444","Type":"ContainerStarted","Data":"4ef3938c532c01b8933f1cf7da8c89d423a8bf3bb81e2be833674da1ae827b05"}
Oct 30 14:47:06.081772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:06.081749    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"
Oct 30 14:47:06.083019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:06.082996    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n" event={"ID":"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e","Type":"ContainerStarted","Data":"1a90a4cd37dc6fd71e2255c7a92a0ebde9841cf8f3e4e048e5c2d93ff9ee28d3"}
Oct 30 14:47:06.131182 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:06.131137    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq" podStartSLOduration=1.131126023 podStartE2EDuration="1.131126023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:06.130590205 +0000 UTC m=+3491.162489143" watchObservedRunningTime="2026-10-30 14:47:06.131126023 +0000 UTC m=+3491.163024959"
Oct 30 14:47:07.093960 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:07.093917    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n" event={"ID":"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e","Type":"ContainerStarted","Data":"cc9ad84cb62bc390929cf9c492a4c92d426ddf851ef0c1991d4ec4dd3d278dd2"}
Oct 30 14:47:07.094463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:07.094017    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"
Oct 30 14:47:07.145170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:07.145111    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n" podStartSLOduration=2.1450925339999998 podStartE2EDuration="2.145092534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:07.141583424 +0000 UTC m=+3492.173482360" watchObservedRunningTime="2026-10-30 14:47:07.145092534 +0000 UTC m=+3492.176991475"
Oct 30 14:47:08.462203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:08.462166    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"]
Oct 30 14:47:08.504488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:08.504453    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"]
Oct 30 14:47:11.368382 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:47:11.368341    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:47:13.462986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:13.462938    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c" podUID="8240e208-5f61-45a4-b0e2-e6298ce7ab8e" containerName="payload-processing" containerID="cri-o://3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692" gracePeriod=25
Oct 30 14:47:13.505285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:13.505231    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g" podUID="52ec6eeb-78f9-4100-8390-c9688a8f94dc" containerName="payload-pre-processing" containerID="cri-o://38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774" gracePeriod=25
Oct 30 14:47:17.097472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:17.097433    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"
Oct 30 14:47:18.103927 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:18.103897    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"
Oct 30 14:47:21.554514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:21.554469    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g" podUID="52ec6eeb-78f9-4100-8390-c9688a8f94dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:47:21.555054 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:21.554628    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c" podUID="8240e208-5f61-45a4-b0e2-e6298ce7ab8e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:47:31.554340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:31.554296    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c" podUID="8240e208-5f61-45a4-b0e2-e6298ce7ab8e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:47:31.554781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:31.554314    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g" podUID="52ec6eeb-78f9-4100-8390-c9688a8f94dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:47:32.504225 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:32.504187    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:47:33.843133 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:33.843106    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"
Oct 30 14:47:33.866259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:33.866233    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"
Oct 30 14:47:33.964245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:33.964205    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-kube-api-access-xdrj4\" (UniqueName: \"kubernetes.io/projected/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-kube-api-access-xdrj4\") pod \"8240e208-5f61-45a4-b0e2-e6298ce7ab8e\" (UID: \"8240e208-5f61-45a4-b0e2-e6298ce7ab8e\") "
Oct 30 14:47:33.964436 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:33.964340    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-plugins-config-volume\") pod \"8240e208-5f61-45a4-b0e2-e6298ce7ab8e\" (UID: \"8240e208-5f61-45a4-b0e2-e6298ce7ab8e\") "
Oct 30 14:47:33.964510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:33.964448    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ec6eeb-78f9-4100-8390-c9688a8f94dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ec6eeb-78f9-4100-8390-c9688a8f94dc-plugins-config-volume\") pod \"52ec6eeb-78f9-4100-8390-c9688a8f94dc\" (UID: \"52ec6eeb-78f9-4100-8390-c9688a8f94dc\") "
Oct 30 14:47:33.964510 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:33.964479    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ec6eeb-78f9-4100-8390-c9688a8f94dc-kube-api-access-zb9vt\" (UniqueName: \"kubernetes.io/projected/52ec6eeb-78f9-4100-8390-c9688a8f94dc-kube-api-access-zb9vt\") pod \"52ec6eeb-78f9-4100-8390-c9688a8f94dc\" (UID: \"52ec6eeb-78f9-4100-8390-c9688a8f94dc\") "
Oct 30 14:47:33.964864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:33.964835    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-plugins-config-volume" pod "8240e208-5f61-45a4-b0e2-e6298ce7ab8e" (UID: "8240e208-5f61-45a4-b0e2-e6298ce7ab8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:47:33.965178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:33.965143    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ec6eeb-78f9-4100-8390-c9688a8f94dc-plugins-config-volume" pod "52ec6eeb-78f9-4100-8390-c9688a8f94dc" (UID: "52ec6eeb-78f9-4100-8390-c9688a8f94dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:47:33.966982 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:33.966957    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ec6eeb-78f9-4100-8390-c9688a8f94dc-kube-api-access-zb9vt" pod "52ec6eeb-78f9-4100-8390-c9688a8f94dc" (UID: "52ec6eeb-78f9-4100-8390-c9688a8f94dc"). InnerVolumeSpecName "kube-api-access-zb9vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:47:33.967074 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:33.967001    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-kube-api-access-xdrj4" pod "8240e208-5f61-45a4-b0e2-e6298ce7ab8e" (UID: "8240e208-5f61-45a4-b0e2-e6298ce7ab8e"). InnerVolumeSpecName "kube-api-access-xdrj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:47:34.066733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.066649    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:34.066733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.066682    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ec6eeb-78f9-4100-8390-c9688a8f94dc-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:34.066733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.066696    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb9vt\" (UniqueName: \"kubernetes.io/projected/52ec6eeb-78f9-4100-8390-c9688a8f94dc-kube-api-access-zb9vt\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:34.066733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.066708    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdrj4\" (UniqueName: \"kubernetes.io/projected/8240e208-5f61-45a4-b0e2-e6298ce7ab8e-kube-api-access-xdrj4\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:34.322780 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.322698    4530 generic.go:356] "Generic (PLEG): container finished" podID="52ec6eeb-78f9-4100-8390-c9688a8f94dc" containerID="38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774" exitCode=0
Oct 30 14:47:34.322780 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.322742    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"
Oct 30 14:47:34.322977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.322785    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g" event={"ID":"52ec6eeb-78f9-4100-8390-c9688a8f94dc","Type":"ContainerDied","Data":"38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774"}
Oct 30 14:47:34.322977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.322841    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g" event={"ID":"52ec6eeb-78f9-4100-8390-c9688a8f94dc","Type":"ContainerDied","Data":"b81a7e0147f0e9b3acd5332697b8a58607b7deb6d479d51060ddd207514cbc09"}
Oct 30 14:47:34.322977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.322869    4530 scope.go:122] "RemoveContainer" containerID="38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774"
Oct 30 14:47:34.324640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.324615    4530 generic.go:356] "Generic (PLEG): container finished" podID="8240e208-5f61-45a4-b0e2-e6298ce7ab8e" containerID="3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692" exitCode=0
Oct 30 14:47:34.324764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.324661    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"
Oct 30 14:47:34.324764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.324679    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c" event={"ID":"8240e208-5f61-45a4-b0e2-e6298ce7ab8e","Type":"ContainerDied","Data":"3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692"}
Oct 30 14:47:34.324764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.324715    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c" event={"ID":"8240e208-5f61-45a4-b0e2-e6298ce7ab8e","Type":"ContainerDied","Data":"6fb4bdf03a61671046da5815f808399b26d72e253126118917c7dee34a210562"}
Oct 30 14:47:34.335701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.335678    4530 scope.go:122] "RemoveContainer" containerID="38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774"
Oct 30 14:47:34.336017 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:47:34.335995    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774\": container with ID starting with 38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774 not found: ID does not exist" containerID="38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774"
Oct 30 14:47:34.336084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.336025    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774"} err="failed to get container status \"38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774\": rpc error: code = NotFound desc = could not find container \"38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774\": container with ID starting with 38f9653ef3bdaa1333e8b9b5a4f32991f2f7612ec1dece0cf6aae6d6a12a7774 not found: ID does not exist"
Oct 30 14:47:34.336084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.336041    4530 scope.go:122] "RemoveContainer" containerID="3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692"
Oct 30 14:47:34.347446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.347428    4530 scope.go:122] "RemoveContainer" containerID="3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692"
Oct 30 14:47:34.347731 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:47:34.347708    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692\": container with ID starting with 3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692 not found: ID does not exist" containerID="3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692"
Oct 30 14:47:34.347834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.347734    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692"} err="failed to get container status \"3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692\": rpc error: code = NotFound desc = could not find container \"3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692\": container with ID starting with 3b0307bd777f159e57e421fca786761772de5aba9d7d5762d0d77ec76af0c692 not found: ID does not exist"
Oct 30 14:47:34.396498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.396467    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"]
Oct 30 14:47:34.410505 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.410477    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e91df-549c95f675-6zb6g"]
Oct 30 14:47:34.466148 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.466113    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"]
Oct 30 14:47:34.473825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:34.473792    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6e91df-5cc45cccd4-5mt9c"]
Oct 30 14:47:35.119013 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:35.118980    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ec6eeb-78f9-4100-8390-c9688a8f94dc" path="/var/lib/kubelet/pods/52ec6eeb-78f9-4100-8390-c9688a8f94dc/volumes"
Oct 30 14:47:35.119439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:35.119425    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8240e208-5f61-45a4-b0e2-e6298ce7ab8e" path="/var/lib/kubelet/pods/8240e208-5f61-45a4-b0e2-e6298ce7ab8e/volumes"
Oct 30 14:47:40.994962 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:40.994927    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"]
Oct 30 14:47:40.995394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:40.995119    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f" podUID="d4ce80eb-7159-47c6-a5b5-cf94871d1e46" containerName="payload-processing" containerID="cri-o://77306de49e7b8c6a7cc05f6479bcb52a91ca7af813026a29449dca9d6e52f227" gracePeriod=30
Oct 30 14:47:41.033330 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.033295    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"]
Oct 30 14:47:41.033697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.033667    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn" podUID="c0852550-3a42-4c43-9c97-c73ac997c4d7" containerName="payload-pre-processing" containerID="cri-o://42a77c50d27fd74786502cbfb94e0f086897f99cceb6667d65895de8684730b2" gracePeriod=30
Oct 30 14:47:41.536418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.532496    4530 generic.go:356] "Generic (PLEG): container finished" podID="c0852550-3a42-4c43-9c97-c73ac997c4d7" containerID="42a77c50d27fd74786502cbfb94e0f086897f99cceb6667d65895de8684730b2" exitCode=0
Oct 30 14:47:41.536418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.532646    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn" event={"ID":"c0852550-3a42-4c43-9c97-c73ac997c4d7","Type":"ContainerDied","Data":"42a77c50d27fd74786502cbfb94e0f086897f99cceb6667d65895de8684730b2"}
Oct 30 14:47:41.538640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.537796    4530 generic.go:356] "Generic (PLEG): container finished" podID="d4ce80eb-7159-47c6-a5b5-cf94871d1e46" containerID="77306de49e7b8c6a7cc05f6479bcb52a91ca7af813026a29449dca9d6e52f227" exitCode=0
Oct 30 14:47:41.538640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.537848    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f" event={"ID":"d4ce80eb-7159-47c6-a5b5-cf94871d1e46","Type":"ContainerDied","Data":"77306de49e7b8c6a7cc05f6479bcb52a91ca7af813026a29449dca9d6e52f227"}
Oct 30 14:47:41.560040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.560003    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"
Oct 30 14:47:41.561395 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:47:41.561361    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:47:41.562715 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.562694    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"
Oct 30 14:47:41.624313 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.624275    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-plugins-config-volume\") pod \"d4ce80eb-7159-47c6-a5b5-cf94871d1e46\" (UID: \"d4ce80eb-7159-47c6-a5b5-cf94871d1e46\") "
Oct 30 14:47:41.624626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.624601    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0852550-3a42-4c43-9c97-c73ac997c4d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0852550-3a42-4c43-9c97-c73ac997c4d7-plugins-config-volume\") pod \"c0852550-3a42-4c43-9c97-c73ac997c4d7\" (UID: \"c0852550-3a42-4c43-9c97-c73ac997c4d7\") "
Oct 30 14:47:41.624860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.624842    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0852550-3a42-4c43-9c97-c73ac997c4d7-kube-api-access-6qmkd\" (UniqueName: \"kubernetes.io/projected/c0852550-3a42-4c43-9c97-c73ac997c4d7-kube-api-access-6qmkd\") pod \"c0852550-3a42-4c43-9c97-c73ac997c4d7\" (UID: \"c0852550-3a42-4c43-9c97-c73ac997c4d7\") "
Oct 30 14:47:41.625098 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.625080    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-kube-api-access-fwxls\" (UniqueName: \"kubernetes.io/projected/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-kube-api-access-fwxls\") pod \"d4ce80eb-7159-47c6-a5b5-cf94871d1e46\" (UID: \"d4ce80eb-7159-47c6-a5b5-cf94871d1e46\") "
Oct 30 14:47:41.628953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.628914    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0852550-3a42-4c43-9c97-c73ac997c4d7-plugins-config-volume" pod "c0852550-3a42-4c43-9c97-c73ac997c4d7" (UID: "c0852550-3a42-4c43-9c97-c73ac997c4d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:47:41.629380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.629354    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-plugins-config-volume" pod "d4ce80eb-7159-47c6-a5b5-cf94871d1e46" (UID: "d4ce80eb-7159-47c6-a5b5-cf94871d1e46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:47:41.632234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.632205    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0852550-3a42-4c43-9c97-c73ac997c4d7-kube-api-access-6qmkd" pod "c0852550-3a42-4c43-9c97-c73ac997c4d7" (UID: "c0852550-3a42-4c43-9c97-c73ac997c4d7"). InnerVolumeSpecName "kube-api-access-6qmkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:47:41.636184 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.636145    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-kube-api-access-fwxls" pod "d4ce80eb-7159-47c6-a5b5-cf94871d1e46" (UID: "d4ce80eb-7159-47c6-a5b5-cf94871d1e46"). InnerVolumeSpecName "kube-api-access-fwxls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:47:41.727528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.727489    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:41.727528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.727525    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0852550-3a42-4c43-9c97-c73ac997c4d7-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:41.727528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.727534    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qmkd\" (UniqueName: \"kubernetes.io/projected/c0852550-3a42-4c43-9c97-c73ac997c4d7-kube-api-access-6qmkd\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:41.727528 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:41.727563    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwxls\" (UniqueName: \"kubernetes.io/projected/d4ce80eb-7159-47c6-a5b5-cf94871d1e46-kube-api-access-fwxls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:42.546709 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:42.546670    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"
Oct 30 14:47:42.547237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:42.546968    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn" event={"ID":"c0852550-3a42-4c43-9c97-c73ac997c4d7","Type":"ContainerDied","Data":"8a88ff948991fdc43e3069e9331ce4d1a38f081c8b516f3ef53611549ab7e4de"}
Oct 30 14:47:42.547237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:42.547037    4530 scope.go:122] "RemoveContainer" containerID="42a77c50d27fd74786502cbfb94e0f086897f99cceb6667d65895de8684730b2"
Oct 30 14:47:42.549076 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:42.549044    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f" event={"ID":"d4ce80eb-7159-47c6-a5b5-cf94871d1e46","Type":"ContainerDied","Data":"9dbd2405f4401336c920decdb58edeb4c50e9e2e5bc0ba0f69d32531de74b633"}
Oct 30 14:47:42.549217 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:42.549164    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"
Oct 30 14:47:42.559999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:42.559979    4530 scope.go:122] "RemoveContainer" containerID="77306de49e7b8c6a7cc05f6479bcb52a91ca7af813026a29449dca9d6e52f227"
Oct 30 14:47:42.609617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:42.609592    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"]
Oct 30 14:47:42.623654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:42.623630    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-6d979dbf9-jdqzn"]
Oct 30 14:47:42.663131 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:42.663101    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"]
Oct 30 14:47:42.669700 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:42.669668    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-78b48cfc98-7942f"]
Oct 30 14:47:43.116439 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.116404    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0852550-3a42-4c43-9c97-c73ac997c4d7" path="/var/lib/kubelet/pods/c0852550-3a42-4c43-9c97-c73ac997c4d7/volumes"
Oct 30 14:47:43.116851 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.116836    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ce80eb-7159-47c6-a5b5-cf94871d1e46" path="/var/lib/kubelet/pods/d4ce80eb-7159-47c6-a5b5-cf94871d1e46/volumes"
Oct 30 14:47:43.325260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.325224    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"]
Oct 30 14:47:43.325444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.325414    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl" podUID="940bb1b7-7692-4676-8b62-5f283e2b0811" containerName="maas-api" containerID="cri-o://0830240c7a31e65ab8063cd5bddd332117ed31679f15af224f79625ff3c034be" gracePeriod=30
Oct 30 14:47:43.558947 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.558914    4530 generic.go:356] "Generic (PLEG): container finished" podID="940bb1b7-7692-4676-8b62-5f283e2b0811" containerID="0830240c7a31e65ab8063cd5bddd332117ed31679f15af224f79625ff3c034be" exitCode=0
Oct 30 14:47:43.559390 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.558999    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl" event={"ID":"940bb1b7-7692-4676-8b62-5f283e2b0811","Type":"ContainerDied","Data":"0830240c7a31e65ab8063cd5bddd332117ed31679f15af224f79625ff3c034be"}
Oct 30 14:47:43.659506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.659476    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:47:43.797618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.770095    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940bb1b7-7692-4676-8b62-5f283e2b0811-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940bb1b7-7692-4676-8b62-5f283e2b0811-openshift-service-ca\") pod \"940bb1b7-7692-4676-8b62-5f283e2b0811\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") "
Oct 30 14:47:43.797618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.770150    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940bb1b7-7692-4676-8b62-5f283e2b0811-kube-api-access-z62p2\" (UniqueName: \"kubernetes.io/projected/940bb1b7-7692-4676-8b62-5f283e2b0811-kube-api-access-z62p2\") pod \"940bb1b7-7692-4676-8b62-5f283e2b0811\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") "
Oct 30 14:47:43.797618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.770246    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-maas-api-tls\") pod \"940bb1b7-7692-4676-8b62-5f283e2b0811\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") "
Oct 30 14:47:43.797618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.770305    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-metrics-certs\" (UniqueName: \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-metrics-certs\") pod \"940bb1b7-7692-4676-8b62-5f283e2b0811\" (UID: \"940bb1b7-7692-4676-8b62-5f283e2b0811\") "
Oct 30 14:47:43.822840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.822776    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940bb1b7-7692-4676-8b62-5f283e2b0811-openshift-service-ca" pod "940bb1b7-7692-4676-8b62-5f283e2b0811" (UID: "940bb1b7-7692-4676-8b62-5f283e2b0811"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:47:43.914864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.896934    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/940bb1b7-7692-4676-8b62-5f283e2b0811-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:43.923138 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.923096    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940bb1b7-7692-4676-8b62-5f283e2b0811-kube-api-access-z62p2" pod "940bb1b7-7692-4676-8b62-5f283e2b0811" (UID: "940bb1b7-7692-4676-8b62-5f283e2b0811"). InnerVolumeSpecName "kube-api-access-z62p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:47:43.933916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.923571    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-maas-api-tls" pod "940bb1b7-7692-4676-8b62-5f283e2b0811" (UID: "940bb1b7-7692-4676-8b62-5f283e2b0811"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:47:43.934323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.934285    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-metrics-certs" pod "940bb1b7-7692-4676-8b62-5f283e2b0811" (UID: "940bb1b7-7692-4676-8b62-5f283e2b0811"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:47:43.998227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.998195    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z62p2\" (UniqueName: \"kubernetes.io/projected/940bb1b7-7692-4676-8b62-5f283e2b0811-kube-api-access-z62p2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:43.998227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.998224    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:43.998227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:43.998235    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/940bb1b7-7692-4676-8b62-5f283e2b0811-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:44.024269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.024236    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"]
Oct 30 14:47:44.024464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.024442    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2" podUID="7aa3d073-f26b-49a3-b32e-25dea19ddf12" containerName="payload-processing" containerID="cri-o://90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197" gracePeriod=30
Oct 30 14:47:44.064940 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.064841    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"]
Oct 30 14:47:44.065121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.065090    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx" podUID="05d6ddc5-2dd7-4148-be66-b0b0ae909d37" containerName="payload-pre-processing" containerID="cri-o://d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053" gracePeriod=30
Oct 30 14:47:44.339072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.339037    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"
Oct 30 14:47:44.474581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.473499    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa3d073-f26b-49a3-b32e-25dea19ddf12-kube-api-access-mxpqh\" (UniqueName: \"kubernetes.io/projected/7aa3d073-f26b-49a3-b32e-25dea19ddf12-kube-api-access-mxpqh\") pod \"7aa3d073-f26b-49a3-b32e-25dea19ddf12\" (UID: \"7aa3d073-f26b-49a3-b32e-25dea19ddf12\") "
Oct 30 14:47:44.474581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.473787    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa3d073-f26b-49a3-b32e-25dea19ddf12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa3d073-f26b-49a3-b32e-25dea19ddf12-plugins-config-volume\") pod \"7aa3d073-f26b-49a3-b32e-25dea19ddf12\" (UID: \"7aa3d073-f26b-49a3-b32e-25dea19ddf12\") "
Oct 30 14:47:44.474852 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.474806    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa3d073-f26b-49a3-b32e-25dea19ddf12-plugins-config-volume" pod "7aa3d073-f26b-49a3-b32e-25dea19ddf12" (UID: "7aa3d073-f26b-49a3-b32e-25dea19ddf12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:47:44.492234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.491857    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"
Oct 30 14:47:44.519797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.519757    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa3d073-f26b-49a3-b32e-25dea19ddf12-kube-api-access-mxpqh" pod "7aa3d073-f26b-49a3-b32e-25dea19ddf12" (UID: "7aa3d073-f26b-49a3-b32e-25dea19ddf12"). InnerVolumeSpecName "kube-api-access-mxpqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:47:44.574502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.574458    4530 generic.go:356] "Generic (PLEG): container finished" podID="7aa3d073-f26b-49a3-b32e-25dea19ddf12" containerID="90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197" exitCode=0
Oct 30 14:47:44.575041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.574603    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"
Oct 30 14:47:44.575041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.574598    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2" event={"ID":"7aa3d073-f26b-49a3-b32e-25dea19ddf12","Type":"ContainerDied","Data":"90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197"}
Oct 30 14:47:44.575041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.574765    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2" event={"ID":"7aa3d073-f26b-49a3-b32e-25dea19ddf12","Type":"ContainerDied","Data":"7231b92ad5de96f9a4c0b97b57ad583aa69aee47c2969108ffe94b03c1cb823c"}
Oct 30 14:47:44.575041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.574807    4530 scope.go:122] "RemoveContainer" containerID="90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197"
Oct 30 14:47:44.575041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.574964    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-kube-api-access-zmlrw\" (UniqueName: \"kubernetes.io/projected/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-kube-api-access-zmlrw\") pod \"05d6ddc5-2dd7-4148-be66-b0b0ae909d37\" (UID: \"05d6ddc5-2dd7-4148-be66-b0b0ae909d37\") "
Oct 30 14:47:44.575341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.575103    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-plugins-config-volume\") pod \"05d6ddc5-2dd7-4148-be66-b0b0ae909d37\" (UID: \"05d6ddc5-2dd7-4148-be66-b0b0ae909d37\") "
Oct 30 14:47:44.575713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.575690    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa3d073-f26b-49a3-b32e-25dea19ddf12-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:44.575833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.575709    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-plugins-config-volume" pod "05d6ddc5-2dd7-4148-be66-b0b0ae909d37" (UID: "05d6ddc5-2dd7-4148-be66-b0b0ae909d37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:47:44.575833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.575718    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxpqh\" (UniqueName: \"kubernetes.io/projected/7aa3d073-f26b-49a3-b32e-25dea19ddf12-kube-api-access-mxpqh\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:44.578657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.578013    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"
Oct 30 14:47:44.578657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.578588    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-kube-api-access-zmlrw" pod "05d6ddc5-2dd7-4148-be66-b0b0ae909d37" (UID: "05d6ddc5-2dd7-4148-be66-b0b0ae909d37"). InnerVolumeSpecName "kube-api-access-zmlrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:47:44.578841 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.578721    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl" event={"ID":"940bb1b7-7692-4676-8b62-5f283e2b0811","Type":"ContainerDied","Data":"88f78076841fefb49dcc1122bb8d128009fc7fbfc8f4910396a250a062e4afcf"}
Oct 30 14:47:44.584151 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.584127    4530 generic.go:356] "Generic (PLEG): container finished" podID="05d6ddc5-2dd7-4148-be66-b0b0ae909d37" containerID="d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053" exitCode=0
Oct 30 14:47:44.584277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.584160    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"
Oct 30 14:47:44.584277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.584189    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx" event={"ID":"05d6ddc5-2dd7-4148-be66-b0b0ae909d37","Type":"ContainerDied","Data":"d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053"}
Oct 30 14:47:44.584277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.584228    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx" event={"ID":"05d6ddc5-2dd7-4148-be66-b0b0ae909d37","Type":"ContainerDied","Data":"f5789ba8706647627a7e0a9f9212537ebfb3796bc7547c2f0237f6aa779b8e8d"}
Oct 30 14:47:44.596213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.596174    4530 scope.go:122] "RemoveContainer" containerID="90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197"
Oct 30 14:47:44.596486 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:47:44.596467    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197\": container with ID starting with 90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197 not found: ID does not exist" containerID="90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197"
Oct 30 14:47:44.596610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.596494    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197"} err="failed to get container status \"90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197\": rpc error: code = NotFound desc = could not find container \"90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197\": container with ID starting with 90e0dc10d5aa79e5cae3d63ad983f34eea31a016d0067a949010764926e3b197 not found: ID does not exist"
Oct 30 14:47:44.596610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.596511    4530 scope.go:122] "RemoveContainer" containerID="0830240c7a31e65ab8063cd5bddd332117ed31679f15af224f79625ff3c034be"
Oct 30 14:47:44.613290 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.613262    4530 scope.go:122] "RemoveContainer" containerID="d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053"
Oct 30 14:47:44.623748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.623728    4530 scope.go:122] "RemoveContainer" containerID="d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053"
Oct 30 14:47:44.624032 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:47:44.624012    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053\": container with ID starting with d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053 not found: ID does not exist" containerID="d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053"
Oct 30 14:47:44.624109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.624037    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053"} err="failed to get container status \"d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053\": rpc error: code = NotFound desc = could not find container \"d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053\": container with ID starting with d2b02106172e0a4dad634f9d4a386e613d6520840fef5412f86779b47c274053 not found: ID does not exist"
Oct 30 14:47:44.648951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.648923    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"]
Oct 30 14:47:44.657664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.657640    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6e91df-5cd8bfbff5-m2xjl"]
Oct 30 14:47:44.676796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.676769    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmlrw\" (UniqueName: \"kubernetes.io/projected/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-kube-api-access-zmlrw\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:44.676796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.676795    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d6ddc5-2dd7-4148-be66-b0b0ae909d37-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:44.699093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.699056    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"]
Oct 30 14:47:44.706187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.706164    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6176c3-65cd57864b-tqgc2"]
Oct 30 14:47:44.745156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.745124    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"]
Oct 30 14:47:44.751465 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:44.751435    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-564c7888cf-h2hqx"]
Oct 30 14:47:45.115648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:45.115614    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d6ddc5-2dd7-4148-be66-b0b0ae909d37" path="/var/lib/kubelet/pods/05d6ddc5-2dd7-4148-be66-b0b0ae909d37/volumes"
Oct 30 14:47:45.116034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:45.116021    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa3d073-f26b-49a3-b32e-25dea19ddf12" path="/var/lib/kubelet/pods/7aa3d073-f26b-49a3-b32e-25dea19ddf12/volumes"
Oct 30 14:47:45.116420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:45.116407    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940bb1b7-7692-4676-8b62-5f283e2b0811" path="/var/lib/kubelet/pods/940bb1b7-7692-4676-8b62-5f283e2b0811/volumes"
Oct 30 14:47:54.377578 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:54.377525    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"]
Oct 30 14:47:54.378128 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:54.377797    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy" containerID="cri-o://a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a" gracePeriod=30
Oct 30 14:47:59.656204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.656132    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:47:59.703515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.703483    4530 generic.go:356] "Generic (PLEG): container finished" podID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerID="a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a" exitCode=0
Oct 30 14:47:59.703743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.703526    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"
Oct 30 14:47:59.703743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.703571    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" event={"ID":"d57850de-5a13-4f64-9e13-004cbd6eb898","Type":"ContainerDied","Data":"a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a"}
Oct 30 14:47:59.703743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.703629    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j" event={"ID":"d57850de-5a13-4f64-9e13-004cbd6eb898","Type":"ContainerDied","Data":"511f9be78f3dbccb5ba1eedb5424b50885757d7f4317c2787c82f538ea4cd685"}
Oct 30 14:47:59.703743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.703656    4530 scope.go:122] "RemoveContainer" containerID="a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a"
Oct 30 14:47:59.715078 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.715057    4530 scope.go:122] "RemoveContainer" containerID="12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6"
Oct 30 14:47:59.725718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.725697    4530 scope.go:122] "RemoveContainer" containerID="a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a"
Oct 30 14:47:59.726257 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:47:59.726142    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a\": container with ID starting with a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a not found: ID does not exist" containerID="a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a"
Oct 30 14:47:59.726257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.726174    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a"} err="failed to get container status \"a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a\": rpc error: code = NotFound desc = could not find container \"a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a\": container with ID starting with a9848941fcc641d59b121660465bb48f29a1280b604f815f6857ab1a9165fc9a not found: ID does not exist"
Oct 30 14:47:59.726257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.726197    4530 scope.go:122] "RemoveContainer" containerID="12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6"
Oct 30 14:47:59.727152 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:47:59.727092    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6\": container with ID starting with 12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6 not found: ID does not exist" containerID="12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6"
Oct 30 14:47:59.727152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.727122    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6"} err="failed to get container status \"12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6\": rpc error: code = NotFound desc = could not find container \"12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6\": container with ID starting with 12c08ee7b068c1f0dfa829fecc012b87d126c9ae53c750b5808824e5e73d54e6 not found: ID does not exist"
Oct 30 14:47:59.775079 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.775047    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-podinfo\") pod \"d57850de-5a13-4f64-9e13-004cbd6eb898\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") "
Oct 30 14:47:59.775284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.775127    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-certs\") pod \"d57850de-5a13-4f64-9e13-004cbd6eb898\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") "
Oct 30 14:47:59.775284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.775187    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-token\" (UniqueName: \"kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-token\") pod \"d57850de-5a13-4f64-9e13-004cbd6eb898\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") "
Oct 30 14:47:59.775284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.775218    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-credential-socket\") pod \"d57850de-5a13-4f64-9e13-004cbd6eb898\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") "
Oct 30 14:47:59.775284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.775273    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-envoy\") pod \"d57850de-5a13-4f64-9e13-004cbd6eb898\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") "
Oct 30 14:47:59.775515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.775326    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-socket\") pod \"d57850de-5a13-4f64-9e13-004cbd6eb898\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") "
Oct 30 14:47:59.775515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.775390    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-kube-api-access-sxjjb\" (UniqueName: \"kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-kube-api-access-sxjjb\") pod \"d57850de-5a13-4f64-9e13-004cbd6eb898\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") "
Oct 30 14:47:59.775515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.775507    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57850de-5a13-4f64-9e13-004cbd6eb898-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57850de-5a13-4f64-9e13-004cbd6eb898-istiod-ca-cert\") pod \"d57850de-5a13-4f64-9e13-004cbd6eb898\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") "
Oct 30 14:47:59.775704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.775514    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-certs" pod "d57850de-5a13-4f64-9e13-004cbd6eb898" (UID: "d57850de-5a13-4f64-9e13-004cbd6eb898"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:47:59.775704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.775603    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-data\") pod \"d57850de-5a13-4f64-9e13-004cbd6eb898\" (UID: \"d57850de-5a13-4f64-9e13-004cbd6eb898\") "
Oct 30 14:47:59.776164 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.776140    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:59.776283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.776219    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57850de-5a13-4f64-9e13-004cbd6eb898-istiod-ca-cert" pod "d57850de-5a13-4f64-9e13-004cbd6eb898" (UID: "d57850de-5a13-4f64-9e13-004cbd6eb898"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:47:59.776562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.776448    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-socket" pod "d57850de-5a13-4f64-9e13-004cbd6eb898" (UID: "d57850de-5a13-4f64-9e13-004cbd6eb898"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:47:59.776562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.776475    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-data" pod "d57850de-5a13-4f64-9e13-004cbd6eb898" (UID: "d57850de-5a13-4f64-9e13-004cbd6eb898"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:47:59.776562 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.776518    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-credential-socket" pod "d57850de-5a13-4f64-9e13-004cbd6eb898" (UID: "d57850de-5a13-4f64-9e13-004cbd6eb898"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:47:59.777986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.777964    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-podinfo" pod "d57850de-5a13-4f64-9e13-004cbd6eb898" (UID: "d57850de-5a13-4f64-9e13-004cbd6eb898"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:47:59.778093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.778036    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-token" pod "d57850de-5a13-4f64-9e13-004cbd6eb898" (UID: "d57850de-5a13-4f64-9e13-004cbd6eb898"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:47:59.778176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.778151    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-kube-api-access-sxjjb" pod "d57850de-5a13-4f64-9e13-004cbd6eb898" (UID: "d57850de-5a13-4f64-9e13-004cbd6eb898"). InnerVolumeSpecName "kube-api-access-sxjjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:47:59.778249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.778225    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-envoy" pod "d57850de-5a13-4f64-9e13-004cbd6eb898" (UID: "d57850de-5a13-4f64-9e13-004cbd6eb898"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:47:59.877583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.877520    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57850de-5a13-4f64-9e13-004cbd6eb898-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:59.877583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.877581    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:59.877583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.877590    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:59.877583 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.877598    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:59.877861 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.877608    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:59.877861 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.877617    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:59.877861 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.877626    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57850de-5a13-4f64-9e13-004cbd6eb898-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:47:59.877861 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:47:59.877634    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxjjb\" (UniqueName: \"kubernetes.io/projected/d57850de-5a13-4f64-9e13-004cbd6eb898-kube-api-access-sxjjb\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:00.060387 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:00.060352    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"]
Oct 30 14:48:00.074054 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:00.074022    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6176c3-openshift-default-64ddffcc7f-cl55j"]
Oct 30 14:48:01.115263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:01.115222    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" path="/var/lib/kubelet/pods/d57850de-5a13-4f64-9e13-004cbd6eb898/volumes"
Oct 30 14:48:03.345942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:03.345854    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"]
Oct 30 14:48:03.346415 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:03.346116    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerName="istio-proxy" containerID="cri-o://ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c" gracePeriod=30
Oct 30 14:48:08.615706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.615643    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:48:08.705602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.705562    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-kube-api-access-ffcmt\" (UniqueName: \"kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-kube-api-access-ffcmt\") pod \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") "
Oct 30 14:48:08.705830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.705616    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istiod-ca-cert\") pod \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") "
Oct 30 14:48:08.705830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.705669    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-podinfo\") pod \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") "
Oct 30 14:48:08.705830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.705748    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-data\") pod \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") "
Oct 30 14:48:08.705830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.705813    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-token\" (UniqueName: \"kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-token\") pod \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") "
Oct 30 14:48:08.706058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.705856    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-certs\") pod \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") "
Oct 30 14:48:08.706058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.705895    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-socket\") pod \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") "
Oct 30 14:48:08.706058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.705924    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-envoy\") pod \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") "
Oct 30 14:48:08.706058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.705968    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-credential-socket\") pod \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\" (UID: \"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3\") "
Oct 30 14:48:08.706639 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.706610    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-data" pod "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" (UID: "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:08.706817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.706661    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-certs" pod "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" (UID: "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:08.706950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.706682    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-socket" pod "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" (UID: "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:08.707020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.706898    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-credential-socket" pod "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" (UID: "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:08.707020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.706997    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istiod-ca-cert" pod "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" (UID: "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:08.708871 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.708844    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-kube-api-access-ffcmt" pod "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" (UID: "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3"). InnerVolumeSpecName "kube-api-access-ffcmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:08.709013 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.708967    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-envoy" pod "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" (UID: "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:08.709013 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.708983    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-token" pod "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" (UID: "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:08.709370 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.709332    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-podinfo" pod "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" (UID: "6bf51e1b-998e-4a06-aecd-80a7d28f7ab3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:48:08.773340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.773305    4530 generic.go:356] "Generic (PLEG): container finished" podID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerID="ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c" exitCode=0
Oct 30 14:48:08.773534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.773360    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"
Oct 30 14:48:08.773534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.773386    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" event={"ID":"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3","Type":"ContainerDied","Data":"ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c"}
Oct 30 14:48:08.773534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.773447    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc" event={"ID":"6bf51e1b-998e-4a06-aecd-80a7d28f7ab3","Type":"ContainerDied","Data":"85ef7fe3709211b0e89beca031b54c504d28c526c5b4d6ae604ecb78addc2224"}
Oct 30 14:48:08.773534 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.773478    4530 scope.go:122] "RemoveContainer" containerID="ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c"
Oct 30 14:48:08.786121 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.786102    4530 scope.go:122] "RemoveContainer" containerID="6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4"
Oct 30 14:48:08.796199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.796177    4530 scope.go:122] "RemoveContainer" containerID="ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c"
Oct 30 14:48:08.796463 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:08.796442    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c\": container with ID starting with ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c not found: ID does not exist" containerID="ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c"
Oct 30 14:48:08.796529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.796473    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c"} err="failed to get container status \"ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c\": rpc error: code = NotFound desc = could not find container \"ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c\": container with ID starting with ee72dad3a74fafb15dcf6100fc779ed7460af9f705621cd72ec26f87af36e08c not found: ID does not exist"
Oct 30 14:48:08.796529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.796491    4530 scope.go:122] "RemoveContainer" containerID="6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4"
Oct 30 14:48:08.796776 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:08.796755    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4\": container with ID starting with 6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4 not found: ID does not exist" containerID="6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4"
Oct 30 14:48:08.796851 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.796786    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4"} err="failed to get container status \"6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4\": rpc error: code = NotFound desc = could not find container \"6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4\": container with ID starting with 6eeafa6b1f7eb1d61396c14d0a44883b34b4c8f3b57a1d117525f833b35ff3a4 not found: ID does not exist"
Oct 30 14:48:08.807708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.807685    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffcmt\" (UniqueName: \"kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-kube-api-access-ffcmt\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:08.807708 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.807709    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:08.807838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.807718    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:08.807838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.807729    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:08.807838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.807741    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:08.807838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.807753    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:08.807838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.807761    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:08.807838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.807770    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:08.807838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.807778    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:08.846724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.846697    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"]
Oct 30 14:48:08.878417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.878355    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-g9fjc"]
Oct 30 14:48:08.930893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:08.930852    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"]
Oct 30 14:48:09.116265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:09.116232    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" path="/var/lib/kubelet/pods/6bf51e1b-998e-4a06-aecd-80a7d28f7ab3/volumes"
Oct 30 14:48:11.367990 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:11.367952    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:48:11.514601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.514566    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"]
Oct 30 14:48:11.514840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.514780    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7" podUID="e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9" containerName="maas-api" containerID="cri-o://91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd" gracePeriod=30
Oct 30 14:48:11.772380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.772346    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"]
Oct 30 14:48:11.772854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.772802    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs" podUID="f59a9fe9-5a43-4fb8-8446-6b13af4457e9" containerName="payload-pre-processing" containerID="cri-o://59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2" gracePeriod=30
Oct 30 14:48:11.786456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.786380    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:48:11.804338 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.804311    4530 generic.go:356] "Generic (PLEG): container finished" podID="e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9" containerID="91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd" exitCode=0
Oct 30 14:48:11.804457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.804359    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"
Oct 30 14:48:11.804580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.804551    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7" event={"ID":"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9","Type":"ContainerDied","Data":"91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd"}
Oct 30 14:48:11.805409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.804600    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7" event={"ID":"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9","Type":"ContainerDied","Data":"9237d4d881836719668103fdb14453400e68ca740475895d96f68cc4f7b9c43a"}
Oct 30 14:48:11.805409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.804629    4530 scope.go:122] "RemoveContainer" containerID="91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd"
Oct 30 14:48:11.891959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.891915    4530 scope.go:122] "RemoveContainer" containerID="91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd"
Oct 30 14:48:11.892354 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:11.892324    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd\": container with ID starting with 91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd not found: ID does not exist" containerID="91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd"
Oct 30 14:48:11.892484 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.892365    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd"} err="failed to get container status \"91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd\": rpc error: code = NotFound desc = could not find container \"91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd\": container with ID starting with 91a340d29422e6ed28f6860702a09c1008514a774d54e132b50660d3034ab5fd not found: ID does not exist"
Oct 30 14:48:11.944097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.942234    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-maas-api-tls\") pod \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") "
Oct 30 14:48:11.944097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.943246    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-openshift-service-ca\") pod \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") "
Oct 30 14:48:11.944097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.943300    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-metrics-certs\") pod \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") "
Oct 30 14:48:11.944097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.943325    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-kube-api-access-tp599\" (UniqueName: \"kubernetes.io/projected/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-kube-api-access-tp599\") pod \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\" (UID: \"e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9\") "
Oct 30 14:48:11.945942 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.945867    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-maas-api-tls" pod "e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9" (UID: "e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:11.946183 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.946156    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-kube-api-access-tp599" pod "e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9" (UID: "e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9"). InnerVolumeSpecName "kube-api-access-tp599". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:11.946598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.946577    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-openshift-service-ca" pod "e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9" (UID: "e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:11.946734 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:11.946648    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-metrics-certs" pod "e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9" (UID: "e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:12.045419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.045161    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:12.045419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.045192    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:12.045419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.045206    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp599\" (UniqueName: \"kubernetes.io/projected/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-kube-api-access-tp599\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:12.045419 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.045218    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:12.046986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.046959    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"
Oct 30 14:48:12.146283 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.146187    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-kube-api-access-qh9f7\" (UniqueName: \"kubernetes.io/projected/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-kube-api-access-qh9f7\") pod \"f59a9fe9-5a43-4fb8-8446-6b13af4457e9\" (UID: \"f59a9fe9-5a43-4fb8-8446-6b13af4457e9\") "
Oct 30 14:48:12.146640 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.146615    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-plugins-config-volume\") pod \"f59a9fe9-5a43-4fb8-8446-6b13af4457e9\" (UID: \"f59a9fe9-5a43-4fb8-8446-6b13af4457e9\") "
Oct 30 14:48:12.158849 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.158805    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-plugins-config-volume" pod "f59a9fe9-5a43-4fb8-8446-6b13af4457e9" (UID: "f59a9fe9-5a43-4fb8-8446-6b13af4457e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:12.172044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.167708    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-kube-api-access-qh9f7" pod "f59a9fe9-5a43-4fb8-8446-6b13af4457e9" (UID: "f59a9fe9-5a43-4fb8-8446-6b13af4457e9"). InnerVolumeSpecName "kube-api-access-qh9f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:12.249116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.249076    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh9f7\" (UniqueName: \"kubernetes.io/projected/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-kube-api-access-qh9f7\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:12.249116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.249108    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59a9fe9-5a43-4fb8-8446-6b13af4457e9-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:12.261143 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.261113    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"]
Oct 30 14:48:12.264739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.264714    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d05e21-6979ddcbd5-rb9j7"]
Oct 30 14:48:12.814360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.814326    4530 generic.go:356] "Generic (PLEG): container finished" podID="f59a9fe9-5a43-4fb8-8446-6b13af4457e9" containerID="59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2" exitCode=0
Oct 30 14:48:12.814885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.814370    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs" event={"ID":"f59a9fe9-5a43-4fb8-8446-6b13af4457e9","Type":"ContainerDied","Data":"59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2"}
Oct 30 14:48:12.814885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.814381    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"
Oct 30 14:48:12.814885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.814420    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs" event={"ID":"f59a9fe9-5a43-4fb8-8446-6b13af4457e9","Type":"ContainerDied","Data":"ea2f4f181dd20327651b59e996138af5281f716bd4389a1940880884ece7edc7"}
Oct 30 14:48:12.814885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.814452    4530 scope.go:122] "RemoveContainer" containerID="59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2"
Oct 30 14:48:12.825617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.825599    4530 scope.go:122] "RemoveContainer" containerID="59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2"
Oct 30 14:48:12.825861 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:12.825844    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2\": container with ID starting with 59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2 not found: ID does not exist" containerID="59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2"
Oct 30 14:48:12.825930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.825869    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2"} err="failed to get container status \"59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2\": rpc error: code = NotFound desc = could not find container \"59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2\": container with ID starting with 59c25b0910be7ab505c8f89bbc499678e3b66db2738ab209e8e6a1b1eae3f9c2 not found: ID does not exist"
Oct 30 14:48:12.891418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.891374    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"]
Oct 30 14:48:12.905443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:12.905414    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5d7d5cc767-4j9qs"]
Oct 30 14:48:13.116053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:13.115976    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9" path="/var/lib/kubelet/pods/e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9/volumes"
Oct 30 14:48:13.116427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:13.116412    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59a9fe9-5a43-4fb8-8446-6b13af4457e9" path="/var/lib/kubelet/pods/f59a9fe9-5a43-4fb8-8446-6b13af4457e9/volumes"
Oct 30 14:48:14.800819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:14.800785    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:48:20.434989 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:20.434949    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"]
Oct 30 14:48:20.435675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:20.435203    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv" podUID="9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" containerName="istio-proxy" containerID="cri-o://06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d" gracePeriod=30
Oct 30 14:48:20.478645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:20.478613    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"]
Oct 30 14:48:20.478833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:20.478811    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh" podUID="34398947-0d72-4b23-b8f8-e61dd6956a1a" containerName="istio-proxy" containerID="cri-o://afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4" gracePeriod=30
Oct 30 14:48:23.954104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:23.954048    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" containerID="cri-o://fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4" gracePeriod=45
Oct 30 14:48:24.088963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.088919    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": dial tcp 10.133.0.95:8000: connect: connection refused"
Oct 30 14:48:24.230952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.230924    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:48:24.303434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.302385    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tls-certs\" (UniqueName: \"kubernetes.io/secret/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tls-certs\") pod \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") "
Oct 30 14:48:24.303434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.302506    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-model-cache\") pod \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") "
Oct 30 14:48:24.303434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.302715    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-home\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-home\") pod \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") "
Oct 30 14:48:24.303434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.302788    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-kube-api-access-8hsn2\" (UniqueName: \"kubernetes.io/projected/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-kube-api-access-8hsn2\") pod \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") "
Oct 30 14:48:24.303434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.302836    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tmp-dir\") pod \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") "
Oct 30 14:48:24.303434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.302900    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-dshm\") pod \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\" (UID: \"276cdf23-88b0-4fc2-a20c-c5c4c27db30f\") "
Oct 30 14:48:24.303434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.302958    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-model-cache" pod "276cdf23-88b0-4fc2-a20c-c5c4c27db30f" (UID: "276cdf23-88b0-4fc2-a20c-c5c4c27db30f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:24.303434 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.303400    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:24.304054 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.303618    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tmp-dir" pod "276cdf23-88b0-4fc2-a20c-c5c4c27db30f" (UID: "276cdf23-88b0-4fc2-a20c-c5c4c27db30f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:24.304054 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.303957    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-home" pod "276cdf23-88b0-4fc2-a20c-c5c4c27db30f" (UID: "276cdf23-88b0-4fc2-a20c-c5c4c27db30f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:24.306716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.305687    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tls-certs" pod "276cdf23-88b0-4fc2-a20c-c5c4c27db30f" (UID: "276cdf23-88b0-4fc2-a20c-c5c4c27db30f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:24.306716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.306192    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-kube-api-access-8hsn2" pod "276cdf23-88b0-4fc2-a20c-c5c4c27db30f" (UID: "276cdf23-88b0-4fc2-a20c-c5c4c27db30f"). InnerVolumeSpecName "kube-api-access-8hsn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:24.306716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.306680    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-dshm" pod "276cdf23-88b0-4fc2-a20c-c5c4c27db30f" (UID: "276cdf23-88b0-4fc2-a20c-c5c4c27db30f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:24.404808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.404772    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:24.404808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.404806    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hsn2\" (UniqueName: \"kubernetes.io/projected/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-kube-api-access-8hsn2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:24.404808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.404821    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:24.405191 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.404834    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:24.405191 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.404845    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276cdf23-88b0-4fc2-a20c-c5c4c27db30f-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:24.901721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.901685    4530 generic.go:356] "Generic (PLEG): container finished" podID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerID="fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4" exitCode=0
Oct 30 14:48:24.901907 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.901728    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"
Oct 30 14:48:24.901907 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.901753    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" event={"ID":"276cdf23-88b0-4fc2-a20c-c5c4c27db30f","Type":"ContainerDied","Data":"fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4"}
Oct 30 14:48:24.901907 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.901786    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8" event={"ID":"276cdf23-88b0-4fc2-a20c-c5c4c27db30f","Type":"ContainerDied","Data":"1f3dbf0471a79e28717f1354174224a6ba19ae5d826d0dde0628b80c82802ceb"}
Oct 30 14:48:24.901907 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.901802    4530 scope.go:122] "RemoveContainer" containerID="fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4"
Oct 30 14:48:24.917780 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.917754    4530 scope.go:122] "RemoveContainer" containerID="fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4"
Oct 30 14:48:24.918158 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:24.918136    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4\": container with ID starting with fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4 not found: ID does not exist" containerID="fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4"
Oct 30 14:48:24.918230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.918168    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4"} err="failed to get container status \"fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4\": rpc error: code = NotFound desc = could not find container \"fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4\": container with ID starting with fd1fcc03a2e9c2ac27fe90a8612124357b1291fa8bbf900026d2ca420473ccd4 not found: ID does not exist"
Oct 30 14:48:24.964930 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.964898    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"]
Oct 30 14:48:24.978639 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:24.978609    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/distinct-w2-da78c4-kserve-5bf54cc768-v7pk8"]
Oct 30 14:48:25.115196 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.115167    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" path="/var/lib/kubelet/pods/276cdf23-88b0-4fc2-a20c-c5c4c27db30f/volumes"
Oct 30 14:48:25.747801 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.747764    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:48:25.784446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.784419    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.823844    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-certs\") pod \"34398947-0d72-4b23-b8f8-e61dd6956a1a\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.823897    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-envoy\") pod \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824003    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-credential-socket\") pod \"34398947-0d72-4b23-b8f8-e61dd6956a1a\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824035    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istiod-ca-cert\") pod \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824068    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-kube-api-access-nk8gg\" (UniqueName: \"kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-kube-api-access-nk8gg\") pod \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824102    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-socket\") pod \"34398947-0d72-4b23-b8f8-e61dd6956a1a\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824125    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-certs" pod "34398947-0d72-4b23-b8f8-e61dd6956a1a" (UID: "34398947-0d72-4b23-b8f8-e61dd6956a1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824138    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-kube-api-access-f4h5c\" (UniqueName: \"kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-kube-api-access-f4h5c\") pod \"34398947-0d72-4b23-b8f8-e61dd6956a1a\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824180    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-certs\") pod \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824226    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-data\") pod \"34398947-0d72-4b23-b8f8-e61dd6956a1a\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824268    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-socket\") pod \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824319    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-podinfo\") pod \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824349    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-credential-socket\") pod \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824432    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-token\" (UniqueName: \"kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-token\") pod \"34398947-0d72-4b23-b8f8-e61dd6956a1a\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824455    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istiod-ca-cert" pod "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" (UID: "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824460    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-envoy\") pod \"34398947-0d72-4b23-b8f8-e61dd6956a1a\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824587    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-podinfo\") pod \"34398947-0d72-4b23-b8f8-e61dd6956a1a\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824659    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-token\" (UniqueName: \"kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-token\") pod \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824691    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34398947-0d72-4b23-b8f8-e61dd6956a1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34398947-0d72-4b23-b8f8-e61dd6956a1a-istiod-ca-cert\") pod \"34398947-0d72-4b23-b8f8-e61dd6956a1a\" (UID: \"34398947-0d72-4b23-b8f8-e61dd6956a1a\") "
Oct 30 14:48:25.824790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.824726    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-data\") pod \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\" (UID: \"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1\") "
Oct 30 14:48:25.827173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.825425    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.827173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.825444    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.827173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.825713    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-data" pod "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" (UID: "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:25.827173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.826532    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-credential-socket" pod "34398947-0d72-4b23-b8f8-e61dd6956a1a" (UID: "34398947-0d72-4b23-b8f8-e61dd6956a1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:25.827173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.826876    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34398947-0d72-4b23-b8f8-e61dd6956a1a-istiod-ca-cert" pod "34398947-0d72-4b23-b8f8-e61dd6956a1a" (UID: "34398947-0d72-4b23-b8f8-e61dd6956a1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:25.827173 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.827139    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-credential-socket" pod "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" (UID: "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:25.827526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.827327    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-socket" pod "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" (UID: "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:25.827614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.827560    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-data" pod "34398947-0d72-4b23-b8f8-e61dd6956a1a" (UID: "34398947-0d72-4b23-b8f8-e61dd6956a1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:25.827614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.827572    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-socket" pod "34398947-0d72-4b23-b8f8-e61dd6956a1a" (UID: "34398947-0d72-4b23-b8f8-e61dd6956a1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:25.828474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.828437    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-certs" pod "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" (UID: "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:25.829766 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.829737    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-envoy" pod "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" (UID: "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:25.835941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.835910    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-podinfo" pod "34398947-0d72-4b23-b8f8-e61dd6956a1a" (UID: "34398947-0d72-4b23-b8f8-e61dd6956a1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:48:25.835941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.835924    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-podinfo" pod "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" (UID: "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:48:25.835941 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.835935    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-kube-api-access-nk8gg" pod "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" (UID: "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1"). InnerVolumeSpecName "kube-api-access-nk8gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:25.836120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.835911    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-envoy" pod "34398947-0d72-4b23-b8f8-e61dd6956a1a" (UID: "34398947-0d72-4b23-b8f8-e61dd6956a1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:25.836120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.835952    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-token" pod "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" (UID: "9a82d8a8-0574-4500-80f1-6fdd38b4c8b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:25.836120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.835967    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-token" pod "34398947-0d72-4b23-b8f8-e61dd6956a1a" (UID: "34398947-0d72-4b23-b8f8-e61dd6956a1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:25.838206 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.838170    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-kube-api-access-f4h5c" pod "34398947-0d72-4b23-b8f8-e61dd6956a1a" (UID: "34398947-0d72-4b23-b8f8-e61dd6956a1a"). InnerVolumeSpecName "kube-api-access-f4h5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:25.911089 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.911049    4530 generic.go:356] "Generic (PLEG): container finished" podID="9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" containerID="06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d" exitCode=0
Oct 30 14:48:25.911248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.911128    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"
Oct 30 14:48:25.911248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.911142    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv" event={"ID":"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1","Type":"ContainerDied","Data":"06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d"}
Oct 30 14:48:25.911248 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.911231    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv" event={"ID":"9a82d8a8-0574-4500-80f1-6fdd38b4c8b1","Type":"ContainerDied","Data":"11dea62eaddbbb0c7b518ca20fe9f320b0a251feee6a5b042236b09a27ef2a90"}
Oct 30 14:48:25.911447 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.911262    4530 scope.go:122] "RemoveContainer" containerID="06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d"
Oct 30 14:48:25.914117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.914088    4530 generic.go:356] "Generic (PLEG): container finished" podID="34398947-0d72-4b23-b8f8-e61dd6956a1a" containerID="afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4" exitCode=0
Oct 30 14:48:25.914204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.914164    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh" event={"ID":"34398947-0d72-4b23-b8f8-e61dd6956a1a","Type":"ContainerDied","Data":"afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4"}
Oct 30 14:48:25.914257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.914207    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh" event={"ID":"34398947-0d72-4b23-b8f8-e61dd6956a1a","Type":"ContainerDied","Data":"6870a1c52fa6e529475704de69b00a34ee94078dcc3f6938433abcee1dea81d7"}
Oct 30 14:48:25.914307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.914292    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"
Oct 30 14:48:25.922816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.922795    4530 scope.go:122] "RemoveContainer" containerID="06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d"
Oct 30 14:48:25.923123 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:25.923091    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d\": container with ID starting with 06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d not found: ID does not exist" containerID="06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d"
Oct 30 14:48:25.923290 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.923254    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d"} err="failed to get container status \"06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d\": rpc error: code = NotFound desc = could not find container \"06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d\": container with ID starting with 06cf210e63a8f063c18fdd9e36c211a51e1a5b15005a16dcaf5254bc17a85b5d not found: ID does not exist"
Oct 30 14:48:25.923290 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.923290    4530 scope.go:122] "RemoveContainer" containerID="afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4"
Oct 30 14:48:25.927298 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927271    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927434    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34398947-0d72-4b23-b8f8-e61dd6956a1a-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927460    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927474    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927488    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927504    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk8gg\" (UniqueName: \"kubernetes.io/projected/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-kube-api-access-nk8gg\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927517    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927532    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4h5c\" (UniqueName: \"kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-kube-api-access-f4h5c\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927570    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927584    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927597    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927611    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927625    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927638    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927652    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.927973 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.927664    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34398947-0d72-4b23-b8f8-e61dd6956a1a-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:25.935175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.935158    4530 scope.go:122] "RemoveContainer" containerID="afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4"
Oct 30 14:48:25.935454 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:25.935431    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4\": container with ID starting with afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4 not found: ID does not exist" containerID="afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4"
Oct 30 14:48:25.935511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:25.935459    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4"} err="failed to get container status \"afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4\": rpc error: code = NotFound desc = could not find container \"afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4\": container with ID starting with afa313bb2225d68825e8f1a6631cdfd5b1ce1e71f12f6988b026d6c4012fc2d4 not found: ID does not exist"
Oct 30 14:48:26.001110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:26.001039    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"]
Oct 30 14:48:26.012811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:26.012786    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-h6lsv"]
Oct 30 14:48:26.082026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:26.081992    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"]
Oct 30 14:48:26.093187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:26.093153    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-xjqjh"]
Oct 30 14:48:27.115486 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:27.115454    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34398947-0d72-4b23-b8f8-e61dd6956a1a" path="/var/lib/kubelet/pods/34398947-0d72-4b23-b8f8-e61dd6956a1a/volumes"
Oct 30 14:48:27.115957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:27.115941    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" path="/var/lib/kubelet/pods/9a82d8a8-0574-4500-80f1-6fdd38b4c8b1/volumes"
Oct 30 14:48:28.684069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:28.684035    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"]
Oct 30 14:48:28.684471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:28.684270    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy" containerID="cri-o://7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f" gracePeriod=30
Oct 30 14:48:31.511645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:31.511608    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"]
Oct 30 14:48:31.547011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:31.546952    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"]
Oct 30 14:48:32.292764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:32.292727    4530 patch_prober.go:29] interesting pod/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:48:32.292952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:32.292786    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:48:33.956375 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:33.956311    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:48:33.977208 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:33.977180    4530 generic.go:356] "Generic (PLEG): container finished" podID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerID="7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f" exitCode=0
Oct 30 14:48:33.977355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:33.977238    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"
Oct 30 14:48:33.977355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:33.977259    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" event={"ID":"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d","Type":"ContainerDied","Data":"7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f"}
Oct 30 14:48:33.977355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:33.977318    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489" event={"ID":"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d","Type":"ContainerDied","Data":"a7335c1c53186b0bce84773ceb8f4936aec9d6433f657b3fb75c8729ee9b9a40"}
Oct 30 14:48:33.977355 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:33.977344    4530 scope.go:122] "RemoveContainer" containerID="7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f"
Oct 30 14:48:33.987495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:33.987477    4530 scope.go:122] "RemoveContainer" containerID="7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649"
Oct 30 14:48:33.997356 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:33.997337    4530 scope.go:122] "RemoveContainer" containerID="7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f"
Oct 30 14:48:33.997715 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:33.997684    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f\": container with ID starting with 7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f not found: ID does not exist" containerID="7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f"
Oct 30 14:48:33.998030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:33.997727    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f"} err="failed to get container status \"7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f\": rpc error: code = NotFound desc = could not find container \"7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f\": container with ID starting with 7490ce9be37e1874a0c67034a4273fe28dea05c4ce8d0bdb84d1d913549bf10f not found: ID does not exist"
Oct 30 14:48:33.998030 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:33.997753    4530 scope.go:122] "RemoveContainer" containerID="7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649"
Oct 30 14:48:33.998848 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:33.998823    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649\": container with ID starting with 7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649 not found: ID does not exist" containerID="7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649"
Oct 30 14:48:33.998970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:33.998851    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649"} err="failed to get container status \"7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649\": rpc error: code = NotFound desc = could not find container \"7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649\": container with ID starting with 7886f201bfc2a24aff9ce77ebf26039fe5ef7c6ee31818a3c4c71265e4344649 not found: ID does not exist"
Oct 30 14:48:34.031422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.031395    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-token\" (UniqueName: \"kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-token\") pod \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") "
Oct 30 14:48:34.031648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.031627    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-data\") pod \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") "
Oct 30 14:48:34.031800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.031679    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-socket\") pod \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") "
Oct 30 14:48:34.031800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.031724    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istiod-ca-cert\") pod \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") "
Oct 30 14:48:34.031800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.031776    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-certs\") pod \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") "
Oct 30 14:48:34.031980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.031802    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-credential-socket\") pod \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") "
Oct 30 14:48:34.031980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.031885    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-kube-api-access-k97wx\" (UniqueName: \"kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-kube-api-access-k97wx\") pod \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") "
Oct 30 14:48:34.031980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.031918    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-envoy\") pod \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") "
Oct 30 14:48:34.031980 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.031949    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-podinfo\") pod \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\" (UID: \"9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d\") "
Oct 30 14:48:34.033227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.032731    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-data" pod "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" (UID: "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:34.033227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.032838    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-certs" pod "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" (UID: "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:34.033227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.032977    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-credential-socket" pod "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" (UID: "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:34.033227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.033175    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-socket" pod "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" (UID: "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:34.033619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.033315    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istiod-ca-cert" pod "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" (UID: "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:34.034402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.034285    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-token" pod "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" (UID: "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:34.035360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.035334    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-podinfo" pod "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" (UID: "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:48:34.035450 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.035384    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-kube-api-access-k97wx" pod "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" (UID: "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d"). InnerVolumeSpecName "kube-api-access-k97wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:34.035579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.035531    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-envoy" pod "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" (UID: "9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:34.133767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.133735    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:34.133767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.133762    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:34.133767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.133772    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:34.134002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.133783    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:34.134002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.133791    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:34.134002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.133799    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k97wx\" (UniqueName: \"kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-kube-api-access-k97wx\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:34.134002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.133808    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:34.134002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.133816    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:34.134002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.133824    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:34.332918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.332884    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"]
Oct 30 14:48:34.347284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:34.347257    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2927fd-openshift-default-64dd8fc7d7-t6489"]
Oct 30 14:48:35.115608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:35.115573    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" path="/var/lib/kubelet/pods/9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d/volumes"
Oct 30 14:48:36.512084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.512041    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf" podUID="06ef73ec-3562-49a8-a436-515403188436" containerName="payload-processing" containerID="cri-o://3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf" gracePeriod=25
Oct 30 14:48:36.548180 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.548142    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f" podUID="8628b21b-fb85-4411-b29f-9b23ef9c05a8" containerName="payload-pre-processing" containerID="cri-o://6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833" gracePeriod=25
Oct 30 14:48:36.820210 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.820178    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"
Oct 30 14:48:36.849391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.849365    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"
Oct 30 14:48:36.868306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.868276    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ef73ec-3562-49a8-a436-515403188436-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ef73ec-3562-49a8-a436-515403188436-plugins-config-volume\") pod \"06ef73ec-3562-49a8-a436-515403188436\" (UID: \"06ef73ec-3562-49a8-a436-515403188436\") "
Oct 30 14:48:36.868501 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.868484    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ef73ec-3562-49a8-a436-515403188436-kube-api-access-kkprq\" (UniqueName: \"kubernetes.io/projected/06ef73ec-3562-49a8-a436-515403188436-kube-api-access-kkprq\") pod \"06ef73ec-3562-49a8-a436-515403188436\" (UID: \"06ef73ec-3562-49a8-a436-515403188436\") "
Oct 30 14:48:36.869146 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.868800    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ef73ec-3562-49a8-a436-515403188436-plugins-config-volume" pod "06ef73ec-3562-49a8-a436-515403188436" (UID: "06ef73ec-3562-49a8-a436-515403188436"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:36.869146 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.869000    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ef73ec-3562-49a8-a436-515403188436-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:36.871876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.871847    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ef73ec-3562-49a8-a436-515403188436-kube-api-access-kkprq" pod "06ef73ec-3562-49a8-a436-515403188436" (UID: "06ef73ec-3562-49a8-a436-515403188436"). InnerVolumeSpecName "kube-api-access-kkprq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:36.970170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.970134    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8628b21b-fb85-4411-b29f-9b23ef9c05a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8628b21b-fb85-4411-b29f-9b23ef9c05a8-plugins-config-volume\") pod \"8628b21b-fb85-4411-b29f-9b23ef9c05a8\" (UID: \"8628b21b-fb85-4411-b29f-9b23ef9c05a8\") "
Oct 30 14:48:36.970343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.970180    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8628b21b-fb85-4411-b29f-9b23ef9c05a8-kube-api-access-z5v7h\" (UniqueName: \"kubernetes.io/projected/8628b21b-fb85-4411-b29f-9b23ef9c05a8-kube-api-access-z5v7h\") pod \"8628b21b-fb85-4411-b29f-9b23ef9c05a8\" (UID: \"8628b21b-fb85-4411-b29f-9b23ef9c05a8\") "
Oct 30 14:48:36.970521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.970496    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8628b21b-fb85-4411-b29f-9b23ef9c05a8-plugins-config-volume" pod "8628b21b-fb85-4411-b29f-9b23ef9c05a8" (UID: "8628b21b-fb85-4411-b29f-9b23ef9c05a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:36.970581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.970507    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkprq\" (UniqueName: \"kubernetes.io/projected/06ef73ec-3562-49a8-a436-515403188436-kube-api-access-kkprq\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:36.972301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:36.972281    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8628b21b-fb85-4411-b29f-9b23ef9c05a8-kube-api-access-z5v7h" pod "8628b21b-fb85-4411-b29f-9b23ef9c05a8" (UID: "8628b21b-fb85-4411-b29f-9b23ef9c05a8"). InnerVolumeSpecName "kube-api-access-z5v7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:37.000229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.000197    4530 generic.go:356] "Generic (PLEG): container finished" podID="8628b21b-fb85-4411-b29f-9b23ef9c05a8" containerID="6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833" exitCode=0
Oct 30 14:48:37.000382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.000248    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"
Oct 30 14:48:37.000382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.000283    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f" event={"ID":"8628b21b-fb85-4411-b29f-9b23ef9c05a8","Type":"ContainerDied","Data":"6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833"}
Oct 30 14:48:37.000382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.000367    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f" event={"ID":"8628b21b-fb85-4411-b29f-9b23ef9c05a8","Type":"ContainerDied","Data":"62edf44f70fd2edfd981387a42a2a3a0b8a9e22f0b39c695c6dfd82a2c14b0ef"}
Oct 30 14:48:37.000521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.000392    4530 scope.go:122] "RemoveContainer" containerID="6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833"
Oct 30 14:48:37.001802 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.001778    4530 generic.go:356] "Generic (PLEG): container finished" podID="06ef73ec-3562-49a8-a436-515403188436" containerID="3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf" exitCode=0
Oct 30 14:48:37.001897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.001827    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"
Oct 30 14:48:37.001978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.001828    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf" event={"ID":"06ef73ec-3562-49a8-a436-515403188436","Type":"ContainerDied","Data":"3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf"}
Oct 30 14:48:37.001978 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.001964    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf" event={"ID":"06ef73ec-3562-49a8-a436-515403188436","Type":"ContainerDied","Data":"e5b7850f47bbb2d3d3c978bbc8f9eff13c1085754ee959623848fc6b3fa2f0f7"}
Oct 30 14:48:37.012408 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.012387    4530 scope.go:122] "RemoveContainer" containerID="6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833"
Oct 30 14:48:37.012709 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:37.012684    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833\": container with ID starting with 6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833 not found: ID does not exist" containerID="6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833"
Oct 30 14:48:37.012789 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.012722    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833"} err="failed to get container status \"6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833\": rpc error: code = NotFound desc = could not find container \"6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833\": container with ID starting with 6549082a853821826adeed808830ac0f556ee37de88bac86a3ac6d8479ba6833 not found: ID does not exist"
Oct 30 14:48:37.012789 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.012740    4530 scope.go:122] "RemoveContainer" containerID="3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf"
Oct 30 14:48:37.022228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.022212    4530 scope.go:122] "RemoveContainer" containerID="3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf"
Oct 30 14:48:37.022502 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:37.022482    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf\": container with ID starting with 3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf not found: ID does not exist" containerID="3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf"
Oct 30 14:48:37.022593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.022509    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf"} err="failed to get container status \"3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf\": rpc error: code = NotFound desc = could not find container \"3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf\": container with ID starting with 3e860a7a880934af114f3d85819fe1ed2369b9fd838bf43ef1d221ca178968cf not found: ID does not exist"
Oct 30 14:48:37.062626 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.062600    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"]
Oct 30 14:48:37.068633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.068610    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-9e3aa0-6f999bc599-dsk2f"]
Oct 30 14:48:37.071734 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.071709    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8628b21b-fb85-4411-b29f-9b23ef9c05a8-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:37.071831 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.071736    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5v7h\" (UniqueName: \"kubernetes.io/projected/8628b21b-fb85-4411-b29f-9b23ef9c05a8-kube-api-access-z5v7h\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:37.116057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.116020    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8628b21b-fb85-4411-b29f-9b23ef9c05a8" path="/var/lib/kubelet/pods/8628b21b-fb85-4411-b29f-9b23ef9c05a8/volumes"
Oct 30 14:48:37.116421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.116404    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"]
Oct 30 14:48:37.116472 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:37.116437    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-9e3aa0-69c745ffb6-qrkwf"]
Oct 30 14:48:39.115608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:39.115572    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ef73ec-3562-49a8-a436-515403188436" path="/var/lib/kubelet/pods/06ef73ec-3562-49a8-a436-515403188436/volumes"
Oct 30 14:48:41.408983 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:41.408938    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:48:43.723029 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:43.722992    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"]
Oct 30 14:48:43.723585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:43.723266    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy" containerID="cri-o://665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a" gracePeriod=30
Oct 30 14:48:49.047033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.047003    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:48:49.167511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.167417    4530 generic.go:356] "Generic (PLEG): container finished" podID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerID="665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a" exitCode=0
Oct 30 14:48:49.167511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.167468    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"
Oct 30 14:48:49.167721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.167505    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" event={"ID":"6a06488b-fc85-45f1-bbf9-a40a39125e5c","Type":"ContainerDied","Data":"665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a"}
Oct 30 14:48:49.167721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.167581    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z" event={"ID":"6a06488b-fc85-45f1-bbf9-a40a39125e5c","Type":"ContainerDied","Data":"0038fc32c4a8a8b1a6e374dc3332c35d002e30f53043112c0f09ee5a82c7cfcf"}
Oct 30 14:48:49.167721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.167611    4530 scope.go:122] "RemoveContainer" containerID="665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a"
Oct 30 14:48:49.178122 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.178102    4530 scope.go:122] "RemoveContainer" containerID="dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612"
Oct 30 14:48:49.178394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.178374    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-kube-api-access-fp64x\" (UniqueName: \"kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-kube-api-access-fp64x\") pod \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") "
Oct 30 14:48:49.178478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.178464    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-data\") pod \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") "
Oct 30 14:48:49.178549 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.178504    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istiod-ca-cert\") pod \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") "
Oct 30 14:48:49.178621 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.178556    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-podinfo\") pod \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") "
Oct 30 14:48:49.178688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.178621    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-certs\") pod \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") "
Oct 30 14:48:49.178688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.178661    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-token\" (UniqueName: \"kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-token\") pod \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") "
Oct 30 14:48:49.178797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.178734    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-credential-socket\") pod \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") "
Oct 30 14:48:49.178797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.178755    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-envoy\") pod \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") "
Oct 30 14:48:49.178797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.178780    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-socket\") pod \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\" (UID: \"6a06488b-fc85-45f1-bbf9-a40a39125e5c\") "
Oct 30 14:48:49.178956 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.178856    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-data" pod "6a06488b-fc85-45f1-bbf9-a40a39125e5c" (UID: "6a06488b-fc85-45f1-bbf9-a40a39125e5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:49.179608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.179048    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-credential-socket" pod "6a06488b-fc85-45f1-bbf9-a40a39125e5c" (UID: "6a06488b-fc85-45f1-bbf9-a40a39125e5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:49.179608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.179295    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-socket" pod "6a06488b-fc85-45f1-bbf9-a40a39125e5c" (UID: "6a06488b-fc85-45f1-bbf9-a40a39125e5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:49.179608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.179316    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-certs" pod "6a06488b-fc85-45f1-bbf9-a40a39125e5c" (UID: "6a06488b-fc85-45f1-bbf9-a40a39125e5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:49.179608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.179323    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istiod-ca-cert" pod "6a06488b-fc85-45f1-bbf9-a40a39125e5c" (UID: "6a06488b-fc85-45f1-bbf9-a40a39125e5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:49.179608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.179493    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:49.179608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.179514    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:49.179608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.179530    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:49.179608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.179564    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:49.179608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.179578    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:49.181035 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.181004    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-podinfo" pod "6a06488b-fc85-45f1-bbf9-a40a39125e5c" (UID: "6a06488b-fc85-45f1-bbf9-a40a39125e5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:48:49.181146 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.181090    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-kube-api-access-fp64x" pod "6a06488b-fc85-45f1-bbf9-a40a39125e5c" (UID: "6a06488b-fc85-45f1-bbf9-a40a39125e5c"). InnerVolumeSpecName "kube-api-access-fp64x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:49.181460 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.181435    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-envoy" pod "6a06488b-fc85-45f1-bbf9-a40a39125e5c" (UID: "6a06488b-fc85-45f1-bbf9-a40a39125e5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:48:49.181614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.181472    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-token" pod "6a06488b-fc85-45f1-bbf9-a40a39125e5c" (UID: "6a06488b-fc85-45f1-bbf9-a40a39125e5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:49.206040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.206020    4530 scope.go:122] "RemoveContainer" containerID="665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a"
Oct 30 14:48:49.206307 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:49.206286    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a\": container with ID starting with 665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a not found: ID does not exist" containerID="665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a"
Oct 30 14:48:49.206352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.206319    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a"} err="failed to get container status \"665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a\": rpc error: code = NotFound desc = could not find container \"665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a\": container with ID starting with 665c363622151daa770b3531657f816bf985e2f903f0529d3ea38bcca5cde90a not found: ID does not exist"
Oct 30 14:48:49.206352 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.206341    4530 scope.go:122] "RemoveContainer" containerID="dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612"
Oct 30 14:48:49.206623 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:49.206603    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612\": container with ID starting with dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612 not found: ID does not exist" containerID="dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612"
Oct 30 14:48:49.206679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.206629    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612"} err="failed to get container status \"dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612\": rpc error: code = NotFound desc = could not find container \"dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612\": container with ID starting with dbce3988050a2b17186c0398e043488975713a163515625f9b3d8896c1b8c612 not found: ID does not exist"
Oct 30 14:48:49.280904 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.280871    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:49.280904 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.280901    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:49.281057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.280911    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a06488b-fc85-45f1-bbf9-a40a39125e5c-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:49.281057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.280920    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp64x\" (UniqueName: \"kubernetes.io/projected/6a06488b-fc85-45f1-bbf9-a40a39125e5c-kube-api-access-fp64x\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:49.521526 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.521488    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"]
Oct 30 14:48:49.535569 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:49.535523    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-kq85z"]
Oct 30 14:48:51.114346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:51.114314    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" path="/var/lib/kubelet/pods/6a06488b-fc85-45f1-bbf9-a40a39125e5c/volumes"
Oct 30 14:48:54.496124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.496081    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"]
Oct 30 14:48:54.496703 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.496362    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq" podUID="9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" containerName="maas-api" containerID="cri-o://60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7" gracePeriod=30
Oct 30 14:48:54.520609 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.520568    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq" podUID="9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.6:8443/health\": dial tcp 10.133.1.6:8443: connect: connection refused"
Oct 30 14:48:54.814315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.814286    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:48:54.906564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.891008    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-maas-api-tls\") pod \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") "
Oct 30 14:48:54.906564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.891071    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-metrics-certs\") pod \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") "
Oct 30 14:48:54.906564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.891321    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-openshift-service-ca\") pod \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") "
Oct 30 14:48:54.906564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.891371    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-kube-api-access-ktskp\" (UniqueName: \"kubernetes.io/projected/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-kube-api-access-ktskp\") pod \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\" (UID: \"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9\") "
Oct 30 14:48:54.906564 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.897977    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-openshift-service-ca" pod "9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" (UID: "9ab4de72-e77d-46aa-afb2-f6cde66ef7b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:54.945939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.945895    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-maas-api-tls" pod "9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" (UID: "9ab4de72-e77d-46aa-afb2-f6cde66ef7b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:54.946160 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.946098    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-metrics-certs" pod "9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" (UID: "9ab4de72-e77d-46aa-afb2-f6cde66ef7b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:54.947679 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.947635    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-kube-api-access-ktskp" pod "9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" (UID: "9ab4de72-e77d-46aa-afb2-f6cde66ef7b9"). InnerVolumeSpecName "kube-api-access-ktskp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:54.993428 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.993391    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:54.993428 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.993424    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktskp\" (UniqueName: \"kubernetes.io/projected/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-kube-api-access-ktskp\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:54.993667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.993440    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:54.993667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.993452    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:54.993667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:54.993499    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:48:55.213453 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:55.213412    4530 generic.go:356] "Generic (PLEG): container finished" podID="9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" containerID="60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7" exitCode=0
Oct 30 14:48:55.213660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:55.213466    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"
Oct 30 14:48:55.213660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:55.213489    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq" event={"ID":"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9","Type":"ContainerDied","Data":"60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7"}
Oct 30 14:48:55.213660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:55.213576    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq" event={"ID":"9ab4de72-e77d-46aa-afb2-f6cde66ef7b9","Type":"ContainerDied","Data":"cfc12daa40f405bbc643faa86902df25d33e1c1632827d34c7a58080c7411f66"}
Oct 30 14:48:55.213660 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:55.213608    4530 scope.go:122] "RemoveContainer" containerID="60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7"
Oct 30 14:48:55.223995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:55.223975    4530 scope.go:122] "RemoveContainer" containerID="02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00"
Oct 30 14:48:55.233522 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:55.233502    4530 scope.go:122] "RemoveContainer" containerID="60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7"
Oct 30 14:48:55.233819 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:55.233792    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7\": container with ID starting with 60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7 not found: ID does not exist" containerID="60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7"
Oct 30 14:48:55.233931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:55.233826    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7"} err="failed to get container status \"60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7\": rpc error: code = NotFound desc = could not find container \"60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7\": container with ID starting with 60bd3419f122706d4fc305872b6838e3c1153bc0e750b0cc38bfda0fdc46fff7 not found: ID does not exist"
Oct 30 14:48:55.233931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:55.233844    4530 scope.go:122] "RemoveContainer" containerID="02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00"
Oct 30 14:48:55.234081 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:55.234063    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00\": container with ID starting with 02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00 not found: ID does not exist" containerID="02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00"
Oct 30 14:48:55.234124 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:55.234088    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00"} err="failed to get container status \"02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00\": rpc error: code = NotFound desc = could not find container \"02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00\": container with ID starting with 02b787705d47a79961254b6116df3344ce042a65f933a9bccc01bb99f5237f00 not found: ID does not exist"
Oct 30 14:48:55.287051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:55.287004    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"]
Oct 30 14:48:55.290782 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:55.290748    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9e3aa0-5894785cd-66phq"]
Oct 30 14:48:56.149378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:56.149349    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:48:56.151623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:56.151595    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:48:56.154950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:56.154931    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:48:56.155971 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:56.155946    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:48:56.157392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:56.157374    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:48:56.158250 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:56.158233    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:48:56.865047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:56.865010    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"]
Oct 30 14:48:56.865335 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:56.865280    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4" podUID="d93cd08a-fef5-441b-9898-799dba609917" containerName="maas-api" containerID="cri-o://26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba" gracePeriod=30
Oct 30 14:48:57.046580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.046524    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"]
Oct 30 14:48:57.046847 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.046818    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n" podUID="a112e79e-e7e5-42c6-a74a-6c63e15fdc1e" containerName="payload-processing" containerID="cri-o://cc9ad84cb62bc390929cf9c492a4c92d426ddf851ef0c1991d4ec4dd3d278dd2" gracePeriod=30
Oct 30 14:48:57.079060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.079024    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"]
Oct 30 14:48:57.079306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.079280    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq" podUID="9223ac67-5eec-4e34-a626-d585e44e5444" containerName="payload-pre-processing" containerID="cri-o://6d18dd94dba7b6f05e4d45b8f7ed6f07cbe2646847e28f79d165e1b92878437e" gracePeriod=30
Oct 30 14:48:57.098574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.098025    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq" podUID="9223ac67-5eec-4e34-a626-d585e44e5444" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.1.55:9004: connect: connection refused"
Oct 30 14:48:57.121817 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.118611    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" path="/var/lib/kubelet/pods/9ab4de72-e77d-46aa-afb2-f6cde66ef7b9/volumes"
Oct 30 14:48:57.213568 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.213523    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:48:57.236818 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.236784    4530 generic.go:356] "Generic (PLEG): container finished" podID="d93cd08a-fef5-441b-9898-799dba609917" containerID="26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba" exitCode=0
Oct 30 14:48:57.236995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.236865    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4" event={"ID":"d93cd08a-fef5-441b-9898-799dba609917","Type":"ContainerDied","Data":"26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba"}
Oct 30 14:48:57.236995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.236916    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4" event={"ID":"d93cd08a-fef5-441b-9898-799dba609917","Type":"ContainerDied","Data":"f4106cbf5467ff4a030a19337665ca598720ab5013d6c6020252e522a3d6d9d4"}
Oct 30 14:48:57.236995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.236937    4530 scope.go:122] "RemoveContainer" containerID="26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba"
Oct 30 14:48:57.237112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.237049    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"
Oct 30 14:48:57.240495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.240465    4530 generic.go:356] "Generic (PLEG): container finished" podID="9223ac67-5eec-4e34-a626-d585e44e5444" containerID="6d18dd94dba7b6f05e4d45b8f7ed6f07cbe2646847e28f79d165e1b92878437e" exitCode=0
Oct 30 14:48:57.240654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.240577    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq" event={"ID":"9223ac67-5eec-4e34-a626-d585e44e5444","Type":"ContainerDied","Data":"6d18dd94dba7b6f05e4d45b8f7ed6f07cbe2646847e28f79d165e1b92878437e"}
Oct 30 14:48:57.243259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.242581    4530 generic.go:356] "Generic (PLEG): container finished" podID="a112e79e-e7e5-42c6-a74a-6c63e15fdc1e" containerID="cc9ad84cb62bc390929cf9c492a4c92d426ddf851ef0c1991d4ec4dd3d278dd2" exitCode=0
Oct 30 14:48:57.243259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.242675    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n" event={"ID":"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e","Type":"ContainerDied","Data":"cc9ad84cb62bc390929cf9c492a4c92d426ddf851ef0c1991d4ec4dd3d278dd2"}
Oct 30 14:48:57.250284 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.250256    4530 scope.go:122] "RemoveContainer" containerID="26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba"
Oct 30 14:48:57.250695 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:48:57.250661    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba\": container with ID starting with 26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba not found: ID does not exist" containerID="26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba"
Oct 30 14:48:57.250790 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.250704    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba"} err="failed to get container status \"26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba\": rpc error: code = NotFound desc = could not find container \"26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba\": container with ID starting with 26be0d253eb374085b72cce76132d7fe713d8dc0b71f67ca9fb6564262033bba not found: ID does not exist"
Oct 30 14:48:57.329214 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.329180    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93cd08a-fef5-441b-9898-799dba609917-kube-api-access-xnnwl\" (UniqueName: \"kubernetes.io/projected/d93cd08a-fef5-441b-9898-799dba609917-kube-api-access-xnnwl\") pod \"d93cd08a-fef5-441b-9898-799dba609917\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") "
Oct 30 14:48:57.329350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.329272    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d93cd08a-fef5-441b-9898-799dba609917-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93cd08a-fef5-441b-9898-799dba609917-openshift-service-ca\") pod \"d93cd08a-fef5-441b-9898-799dba609917\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") "
Oct 30 14:48:57.329350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.329310    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-metrics-certs\") pod \"d93cd08a-fef5-441b-9898-799dba609917\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") "
Oct 30 14:48:57.329491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.329431    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-maas-api-tls\") pod \"d93cd08a-fef5-441b-9898-799dba609917\" (UID: \"d93cd08a-fef5-441b-9898-799dba609917\") "
Oct 30 14:48:57.330838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.330794    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93cd08a-fef5-441b-9898-799dba609917-openshift-service-ca" pod "d93cd08a-fef5-441b-9898-799dba609917" (UID: "d93cd08a-fef5-441b-9898-799dba609917"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:57.333028 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.333001    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-maas-api-tls" pod "d93cd08a-fef5-441b-9898-799dba609917" (UID: "d93cd08a-fef5-441b-9898-799dba609917"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:57.333984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.333959    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-metrics-certs" pod "d93cd08a-fef5-441b-9898-799dba609917" (UID: "d93cd08a-fef5-441b-9898-799dba609917"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:48:57.335302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.335276    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93cd08a-fef5-441b-9898-799dba609917-kube-api-access-xnnwl" pod "d93cd08a-fef5-441b-9898-799dba609917" (UID: "d93cd08a-fef5-441b-9898-799dba609917"). InnerVolumeSpecName "kube-api-access-xnnwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:57.341888 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.341861    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"
Oct 30 14:48:57.428579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.412757    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"
Oct 30 14:48:57.432565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.431479    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-plugins-config-volume\") pod \"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e\" (UID: \"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e\") "
Oct 30 14:48:57.432565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.431587    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-kube-api-access-cq5n9\" (UniqueName: \"kubernetes.io/projected/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-kube-api-access-cq5n9\") pod \"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e\" (UID: \"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e\") "
Oct 30 14:48:57.432565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.432139    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:57.432565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.432160    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnnwl\" (UniqueName: \"kubernetes.io/projected/d93cd08a-fef5-441b-9898-799dba609917-kube-api-access-xnnwl\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:57.432565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.432176    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93cd08a-fef5-441b-9898-799dba609917-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:57.432565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.432189    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93cd08a-fef5-441b-9898-799dba609917-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:57.487787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.440961    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-plugins-config-volume" pod "a112e79e-e7e5-42c6-a74a-6c63e15fdc1e" (UID: "a112e79e-e7e5-42c6-a74a-6c63e15fdc1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:57.487787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.475916    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-kube-api-access-cq5n9" pod "a112e79e-e7e5-42c6-a74a-6c63e15fdc1e" (UID: "a112e79e-e7e5-42c6-a74a-6c63e15fdc1e"). InnerVolumeSpecName "kube-api-access-cq5n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:57.532889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.532851    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9223ac67-5eec-4e34-a626-d585e44e5444-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9223ac67-5eec-4e34-a626-d585e44e5444-plugins-config-volume\") pod \"9223ac67-5eec-4e34-a626-d585e44e5444\" (UID: \"9223ac67-5eec-4e34-a626-d585e44e5444\") "
Oct 30 14:48:57.533127 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.533107    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9223ac67-5eec-4e34-a626-d585e44e5444-kube-api-access-5fwzk\" (UniqueName: \"kubernetes.io/projected/9223ac67-5eec-4e34-a626-d585e44e5444-kube-api-access-5fwzk\") pod \"9223ac67-5eec-4e34-a626-d585e44e5444\" (UID: \"9223ac67-5eec-4e34-a626-d585e44e5444\") "
Oct 30 14:48:57.533312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.533292    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9223ac67-5eec-4e34-a626-d585e44e5444-plugins-config-volume" pod "9223ac67-5eec-4e34-a626-d585e44e5444" (UID: "9223ac67-5eec-4e34-a626-d585e44e5444"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:57.533518 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.533494    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:57.533518 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.533512    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9223ac67-5eec-4e34-a626-d585e44e5444-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:57.533659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.533523    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq5n9\" (UniqueName: \"kubernetes.io/projected/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e-kube-api-access-cq5n9\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:57.535231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.535214    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9223ac67-5eec-4e34-a626-d585e44e5444-kube-api-access-5fwzk" pod "9223ac67-5eec-4e34-a626-d585e44e5444" (UID: "9223ac67-5eec-4e34-a626-d585e44e5444"). InnerVolumeSpecName "kube-api-access-5fwzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:57.596092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.596052    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"]
Oct 30 14:48:57.601687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.601644    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcc710-754856d4f8-vrbn4"]
Oct 30 14:48:57.634845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:57.634773    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fwzk\" (UniqueName: \"kubernetes.io/projected/9223ac67-5eec-4e34-a626-d585e44e5444-kube-api-access-5fwzk\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:58.251619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:58.251588    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"
Oct 30 14:48:58.252126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:58.251614    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq" event={"ID":"9223ac67-5eec-4e34-a626-d585e44e5444","Type":"ContainerDied","Data":"4ef3938c532c01b8933f1cf7da8c89d423a8bf3bb81e2be833674da1ae827b05"}
Oct 30 14:48:58.252126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:58.251696    4530 scope.go:122] "RemoveContainer" containerID="6d18dd94dba7b6f05e4d45b8f7ed6f07cbe2646847e28f79d165e1b92878437e"
Oct 30 14:48:58.253432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:58.253249    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n" event={"ID":"a112e79e-e7e5-42c6-a74a-6c63e15fdc1e","Type":"ContainerDied","Data":"1a90a4cd37dc6fd71e2255c7a92a0ebde9841cf8f3e4e048e5c2d93ff9ee28d3"}
Oct 30 14:48:58.253432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:58.253293    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"
Oct 30 14:48:58.264234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:58.264214    4530 scope.go:122] "RemoveContainer" containerID="cc9ad84cb62bc390929cf9c492a4c92d426ddf851ef0c1991d4ec4dd3d278dd2"
Oct 30 14:48:58.313745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:58.313717    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"]
Oct 30 14:48:58.327047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:58.327020    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-8c75455dc-6s2jq"]
Oct 30 14:48:58.367319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:58.367291    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"]
Oct 30 14:48:58.373521 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:58.373494    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcc710-75c6df84d-h8h7n"]
Oct 30 14:48:59.114868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:59.114835    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9223ac67-5eec-4e34-a626-d585e44e5444" path="/var/lib/kubelet/pods/9223ac67-5eec-4e34-a626-d585e44e5444/volumes"
Oct 30 14:48:59.115257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:59.115244    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a112e79e-e7e5-42c6-a74a-6c63e15fdc1e" path="/var/lib/kubelet/pods/a112e79e-e7e5-42c6-a74a-6c63e15fdc1e/volumes"
Oct 30 14:48:59.115665 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:59.115650    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d93cd08a-fef5-441b-9898-799dba609917" path="/var/lib/kubelet/pods/d93cd08a-fef5-441b-9898-799dba609917/volumes"
Oct 30 14:48:59.464577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:59.464454    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"]
Oct 30 14:48:59.465052 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:59.464702    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s" podUID="c1ab0398-256e-4d59-814a-bde23499d65d" containerName="payload-processing" containerID="cri-o://0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10" gracePeriod=30
Oct 30 14:48:59.740506 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:59.740482    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"
Oct 30 14:48:59.860152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:59.859303    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ab0398-256e-4d59-814a-bde23499d65d-kube-api-access-kgzvm\" (UniqueName: \"kubernetes.io/projected/c1ab0398-256e-4d59-814a-bde23499d65d-kube-api-access-kgzvm\") pod \"c1ab0398-256e-4d59-814a-bde23499d65d\" (UID: \"c1ab0398-256e-4d59-814a-bde23499d65d\") "
Oct 30 14:48:59.860152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:59.859507    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ab0398-256e-4d59-814a-bde23499d65d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ab0398-256e-4d59-814a-bde23499d65d-plugins-config-volume\") pod \"c1ab0398-256e-4d59-814a-bde23499d65d\" (UID: \"c1ab0398-256e-4d59-814a-bde23499d65d\") "
Oct 30 14:48:59.860431 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:59.860200    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ab0398-256e-4d59-814a-bde23499d65d-plugins-config-volume" pod "c1ab0398-256e-4d59-814a-bde23499d65d" (UID: "c1ab0398-256e-4d59-814a-bde23499d65d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:48:59.869600 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:59.867689    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ab0398-256e-4d59-814a-bde23499d65d-kube-api-access-kgzvm" pod "c1ab0398-256e-4d59-814a-bde23499d65d" (UID: "c1ab0398-256e-4d59-814a-bde23499d65d"). InnerVolumeSpecName "kube-api-access-kgzvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:48:59.961058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:59.961022    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgzvm\" (UniqueName: \"kubernetes.io/projected/c1ab0398-256e-4d59-814a-bde23499d65d-kube-api-access-kgzvm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:48:59.961058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:48:59.961049    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ab0398-256e-4d59-814a-bde23499d65d-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:00.285711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:00.285677    4530 generic.go:356] "Generic (PLEG): container finished" podID="c1ab0398-256e-4d59-814a-bde23499d65d" containerID="0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10" exitCode=0
Oct 30 14:49:00.285891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:00.285722    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"
Oct 30 14:49:00.285891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:00.285764    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s" event={"ID":"c1ab0398-256e-4d59-814a-bde23499d65d","Type":"ContainerDied","Data":"0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10"}
Oct 30 14:49:00.285891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:00.285820    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s" event={"ID":"c1ab0398-256e-4d59-814a-bde23499d65d","Type":"ContainerDied","Data":"8c313ec82b9155d64ecb24d22060cd9878a2d1d2b0729d5693f38e760b3fe203"}
Oct 30 14:49:00.285891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:00.285848    4530 scope.go:122] "RemoveContainer" containerID="0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10"
Oct 30 14:49:00.297031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:00.297013    4530 scope.go:122] "RemoveContainer" containerID="0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10"
Oct 30 14:49:00.297295 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:00.297275    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10\": container with ID starting with 0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10 not found: ID does not exist" containerID="0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10"
Oct 30 14:49:00.297371 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:00.297302    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10"} err="failed to get container status \"0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10\": rpc error: code = NotFound desc = could not find container \"0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10\": container with ID starting with 0a6df6dd6d524d56d7d567c554f51e7187ff0ec8827ba55eafd369d82da8fb10 not found: ID does not exist"
Oct 30 14:49:00.385592 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:00.385559    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"]
Oct 30 14:49:00.397015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:00.396984    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-844587bbf8-6rq9s"]
Oct 30 14:49:01.115633 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:01.115595    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ab0398-256e-4d59-814a-bde23499d65d" path="/var/lib/kubelet/pods/c1ab0398-256e-4d59-814a-bde23499d65d/volumes"
Oct 30 14:49:03.795425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:03.795384    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"]
Oct 30 14:49:03.796001 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:03.795647    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95" podUID="85899396-118d-4899-a445-2077f6778fc0" containerName="maas-api" containerID="cri-o://910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644" gracePeriod=30
Oct 30 14:49:04.085767 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.085729    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:49:04.222603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.222530    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-metrics-certs\") pod \"85899396-118d-4899-a445-2077f6778fc0\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") "
Oct 30 14:49:04.222781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.222732    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85899396-118d-4899-a445-2077f6778fc0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85899396-118d-4899-a445-2077f6778fc0-openshift-service-ca\") pod \"85899396-118d-4899-a445-2077f6778fc0\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") "
Oct 30 14:49:04.222827 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.222791    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85899396-118d-4899-a445-2077f6778fc0-kube-api-access-7h7sb\" (UniqueName: \"kubernetes.io/projected/85899396-118d-4899-a445-2077f6778fc0-kube-api-access-7h7sb\") pod \"85899396-118d-4899-a445-2077f6778fc0\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") "
Oct 30 14:49:04.222862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.222842    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-maas-api-tls\") pod \"85899396-118d-4899-a445-2077f6778fc0\" (UID: \"85899396-118d-4899-a445-2077f6778fc0\") "
Oct 30 14:49:04.223165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.223127    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85899396-118d-4899-a445-2077f6778fc0-openshift-service-ca" pod "85899396-118d-4899-a445-2077f6778fc0" (UID: "85899396-118d-4899-a445-2077f6778fc0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:04.223331 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.223292    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85899396-118d-4899-a445-2077f6778fc0-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:04.224859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.224833    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-metrics-certs" pod "85899396-118d-4899-a445-2077f6778fc0" (UID: "85899396-118d-4899-a445-2077f6778fc0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:49:04.225359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.225332    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-maas-api-tls" pod "85899396-118d-4899-a445-2077f6778fc0" (UID: "85899396-118d-4899-a445-2077f6778fc0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:49:04.225446 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.225372    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85899396-118d-4899-a445-2077f6778fc0-kube-api-access-7h7sb" pod "85899396-118d-4899-a445-2077f6778fc0" (UID: "85899396-118d-4899-a445-2077f6778fc0"). InnerVolumeSpecName "kube-api-access-7h7sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:04.318563 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.318472    4530 generic.go:356] "Generic (PLEG): container finished" podID="85899396-118d-4899-a445-2077f6778fc0" containerID="910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644" exitCode=0
Oct 30 14:49:04.318778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.318588    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"
Oct 30 14:49:04.318778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.318608    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95" event={"ID":"85899396-118d-4899-a445-2077f6778fc0","Type":"ContainerDied","Data":"910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644"}
Oct 30 14:49:04.318778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.318662    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95" event={"ID":"85899396-118d-4899-a445-2077f6778fc0","Type":"ContainerDied","Data":"0fd3a2efd0936f732b14fc2345c23080cf44fba08d9259489c18f6187cfd7a1b"}
Oct 30 14:49:04.318778 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.318696    4530 scope.go:122] "RemoveContainer" containerID="910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644"
Oct 30 14:49:04.325091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.324947    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:04.325091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.324980    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h7sb\" (UniqueName: \"kubernetes.io/projected/85899396-118d-4899-a445-2077f6778fc0-kube-api-access-7h7sb\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:04.325091 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.324996    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85899396-118d-4899-a445-2077f6778fc0-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:04.334656 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.334632    4530 scope.go:122] "RemoveContainer" containerID="910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644"
Oct 30 14:49:04.335073 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:04.335038    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644\": container with ID starting with 910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644 not found: ID does not exist" containerID="910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644"
Oct 30 14:49:04.335177 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.335082    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644"} err="failed to get container status \"910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644\": rpc error: code = NotFound desc = could not find container \"910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644\": container with ID starting with 910a672e971fa957c0f95a4fb8bbf7484ad9ee508d7c1cb0f60c70b5825f2644 not found: ID does not exist"
Oct 30 14:49:04.379584 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.379507    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"]
Oct 30 14:49:04.385507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:04.385482    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-da78c4-b85bb8d85-w4m95"]
Oct 30 14:49:05.114048 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:05.114015    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85899396-118d-4899-a445-2077f6778fc0" path="/var/lib/kubelet/pods/85899396-118d-4899-a445-2077f6778fc0/volumes"
Oct 30 14:49:05.808372 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:05.808333    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"]
Oct 30 14:49:05.808598 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:05.808571    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy" containerID="cri-o://56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf" gracePeriod=30
Oct 30 14:49:08.503587 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:08.503522    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"]
Oct 30 14:49:08.504083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:08.503841    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy" containerID="cri-o://f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac" gracePeriod=30
Oct 30 14:49:11.068357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.068325    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:49:11.206332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206250    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-socket\") pod \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") "
Oct 30 14:49:11.206332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206303    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-token\" (UniqueName: \"kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-token\") pod \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") "
Oct 30 14:49:11.206332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206323    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-credential-socket\") pod \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") "
Oct 30 14:49:11.206646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206340    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-envoy\") pod \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") "
Oct 30 14:49:11.206646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206395    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-podinfo\") pod \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") "
Oct 30 14:49:11.206646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206452    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-certs\") pod \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") "
Oct 30 14:49:11.206646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206509    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istiod-ca-cert\") pod \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") "
Oct 30 14:49:11.206646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206572    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-socket" pod "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" (UID: "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:11.206646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206612    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-data\") pod \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") "
Oct 30 14:49:11.206983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206706    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-kube-api-access-tlt7k\" (UniqueName: \"kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-kube-api-access-tlt7k\") pod \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\" (UID: \"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f\") "
Oct 30 14:49:11.206983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206695    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-credential-socket" pod "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" (UID: "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:11.206983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206922    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-certs" pod "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" (UID: "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:11.206983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206935    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-data" pod "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" (UID: "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:11.206983 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.206970    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istiod-ca-cert" pod "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" (UID: "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:11.207291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.207269    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:11.207354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.207296    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:11.207354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.207310    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:11.207354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.207325    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:11.207354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.207339    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:11.208781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.208753    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-podinfo" pod "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" (UID: "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:49:11.208894 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.208779    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-envoy" pod "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" (UID: "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:11.208894 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.208832    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-kube-api-access-tlt7k" pod "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" (UID: "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f"). InnerVolumeSpecName "kube-api-access-tlt7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:11.208894 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.208858    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-token" pod "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" (UID: "d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:11.309017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.308980    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:11.309017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.309011    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlt7k\" (UniqueName: \"kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-kube-api-access-tlt7k\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:11.309017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.309022    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:11.309265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.309030    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:11.369319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.369139    4530 generic.go:356] "Generic (PLEG): container finished" podID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerID="56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf" exitCode=0
Oct 30 14:49:11.369319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.369182    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" event={"ID":"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f","Type":"ContainerDied","Data":"56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf"}
Oct 30 14:49:11.369319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.369212    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"
Oct 30 14:49:11.369319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.369256    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t" event={"ID":"d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f","Type":"ContainerDied","Data":"50c03f4f56fc2435aa86da42618d2e7106ecc0c78072175e71d8751bb5ed9a05"}
Oct 30 14:49:11.369319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.369286    4530 scope.go:122] "RemoveContainer" containerID="56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf"
Oct 30 14:49:11.373090 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:11.372682    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:49:11.385042 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.385019    4530 scope.go:122] "RemoveContainer" containerID="20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1"
Oct 30 14:49:11.394565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.394530    4530 scope.go:122] "RemoveContainer" containerID="56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf"
Oct 30 14:49:11.394862 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:11.394837    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf\": container with ID starting with 56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf not found: ID does not exist" containerID="56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf"
Oct 30 14:49:11.394959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.394872    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf"} err="failed to get container status \"56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf\": rpc error: code = NotFound desc = could not find container \"56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf\": container with ID starting with 56e50a55d9fe2e8eaeb6737cc7e92c73cffa58f47ebe8286d853412d0c6776bf not found: ID does not exist"
Oct 30 14:49:11.394959 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.394890    4530 scope.go:122] "RemoveContainer" containerID="20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1"
Oct 30 14:49:11.395153 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:11.395136    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1\": container with ID starting with 20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1 not found: ID does not exist" containerID="20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1"
Oct 30 14:49:11.395215 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.395157    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1"} err="failed to get container status \"20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1\": rpc error: code = NotFound desc = could not find container \"20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1\": container with ID starting with 20e74dbaa37e1d4d67e95c9a48a088a21490d2985521f7770617b96531294da1 not found: ID does not exist"
Oct 30 14:49:11.443601 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.443561    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"]
Oct 30 14:49:11.449723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:11.449695    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-bcc710-openshift-default-645587584f-q2m2t"]
Oct 30 14:49:12.131170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.118435    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"]
Oct 30 14:49:12.131170 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.118687    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w" podUID="3214e243-6b64-417d-8711-4b3cacb3c1bb" containerName="maas-api" containerID="cri-o://35ad71c79a7ea1ac1085520dba1645871f8da5a06524074c3aed2e76b38a690a" gracePeriod=30
Oct 30 14:49:12.445905 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.445871    4530 generic.go:356] "Generic (PLEG): container finished" podID="3214e243-6b64-417d-8711-4b3cacb3c1bb" containerID="35ad71c79a7ea1ac1085520dba1645871f8da5a06524074c3aed2e76b38a690a" exitCode=0
Oct 30 14:49:12.446087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.445945    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w" event={"ID":"3214e243-6b64-417d-8711-4b3cacb3c1bb","Type":"ContainerDied","Data":"35ad71c79a7ea1ac1085520dba1645871f8da5a06524074c3aed2e76b38a690a"}
Oct 30 14:49:12.446087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.445997    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w" event={"ID":"3214e243-6b64-417d-8711-4b3cacb3c1bb","Type":"ContainerDied","Data":"47efef1e93c0bb4cc142b1368797c3c58cc518facee1db189c5b104a4fa00a30"}
Oct 30 14:49:12.446087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.446015    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47efef1e93c0bb4cc142b1368797c3c58cc518facee1db189c5b104a4fa00a30"
Oct 30 14:49:12.448301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.448276    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:49:12.539697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.539665    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-maas-api-tls\") pod \"3214e243-6b64-417d-8711-4b3cacb3c1bb\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") "
Oct 30 14:49:12.539876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.539774    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3214e243-6b64-417d-8711-4b3cacb3c1bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3214e243-6b64-417d-8711-4b3cacb3c1bb-openshift-service-ca\") pod \"3214e243-6b64-417d-8711-4b3cacb3c1bb\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") "
Oct 30 14:49:12.539876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.539842    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3214e243-6b64-417d-8711-4b3cacb3c1bb-kube-api-access-v679w\" (UniqueName: \"kubernetes.io/projected/3214e243-6b64-417d-8711-4b3cacb3c1bb-kube-api-access-v679w\") pod \"3214e243-6b64-417d-8711-4b3cacb3c1bb\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") "
Oct 30 14:49:12.539995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.539948    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-metrics-certs\") pod \"3214e243-6b64-417d-8711-4b3cacb3c1bb\" (UID: \"3214e243-6b64-417d-8711-4b3cacb3c1bb\") "
Oct 30 14:49:12.540204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.540179    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3214e243-6b64-417d-8711-4b3cacb3c1bb-openshift-service-ca" pod "3214e243-6b64-417d-8711-4b3cacb3c1bb" (UID: "3214e243-6b64-417d-8711-4b3cacb3c1bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:12.540432 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.540408    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3214e243-6b64-417d-8711-4b3cacb3c1bb-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:12.541958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.541933    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-metrics-certs" pod "3214e243-6b64-417d-8711-4b3cacb3c1bb" (UID: "3214e243-6b64-417d-8711-4b3cacb3c1bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:49:12.542037 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.541979    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-maas-api-tls" pod "3214e243-6b64-417d-8711-4b3cacb3c1bb" (UID: "3214e243-6b64-417d-8711-4b3cacb3c1bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:49:12.542199 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.542182    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3214e243-6b64-417d-8711-4b3cacb3c1bb-kube-api-access-v679w" pod "3214e243-6b64-417d-8711-4b3cacb3c1bb" (UID: "3214e243-6b64-417d-8711-4b3cacb3c1bb"). InnerVolumeSpecName "kube-api-access-v679w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:12.642319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.642213    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:12.642319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.642259    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v679w\" (UniqueName: \"kubernetes.io/projected/3214e243-6b64-417d-8711-4b3cacb3c1bb-kube-api-access-v679w\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:12.642319 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:12.642289    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3214e243-6b64-417d-8711-4b3cacb3c1bb-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:13.125567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.122036    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" path="/var/lib/kubelet/pods/d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f/volumes"
Oct 30 14:49:13.454315 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.454230    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"
Oct 30 14:49:13.515255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.515223    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"]
Oct 30 14:49:13.529473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.529446    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e371c3-6dc5bb86f5-gdc4w"]
Oct 30 14:49:13.780067 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.780034    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:49:13.958691 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.958653    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-token\" (UniqueName: \"kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-token\") pod \"55ddd71b-1fbf-4359-b8a3-da951b41869b\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") "
Oct 30 14:49:13.958691 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.958689    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-certs\") pod \"55ddd71b-1fbf-4359-b8a3-da951b41869b\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") "
Oct 30 14:49:13.958917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.958712    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-data\") pod \"55ddd71b-1fbf-4359-b8a3-da951b41869b\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") "
Oct 30 14:49:13.958917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.958741    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-credential-socket\") pod \"55ddd71b-1fbf-4359-b8a3-da951b41869b\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") "
Oct 30 14:49:13.958917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.958761    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-kube-api-access-mjhdb\" (UniqueName: \"kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-kube-api-access-mjhdb\") pod \"55ddd71b-1fbf-4359-b8a3-da951b41869b\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") "
Oct 30 14:49:13.958917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.958802    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-envoy\") pod \"55ddd71b-1fbf-4359-b8a3-da951b41869b\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") "
Oct 30 14:49:13.958917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.958821    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ddd71b-1fbf-4359-b8a3-da951b41869b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ddd71b-1fbf-4359-b8a3-da951b41869b-istiod-ca-cert\") pod \"55ddd71b-1fbf-4359-b8a3-da951b41869b\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") "
Oct 30 14:49:13.958917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.958862    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-socket\") pod \"55ddd71b-1fbf-4359-b8a3-da951b41869b\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") "
Oct 30 14:49:13.959265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.958916    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-podinfo\") pod \"55ddd71b-1fbf-4359-b8a3-da951b41869b\" (UID: \"55ddd71b-1fbf-4359-b8a3-da951b41869b\") "
Oct 30 14:49:13.959265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.959042    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-credential-socket" pod "55ddd71b-1fbf-4359-b8a3-da951b41869b" (UID: "55ddd71b-1fbf-4359-b8a3-da951b41869b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:13.959265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.959054    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-certs" pod "55ddd71b-1fbf-4359-b8a3-da951b41869b" (UID: "55ddd71b-1fbf-4359-b8a3-da951b41869b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:13.959265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.959071    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-data" pod "55ddd71b-1fbf-4359-b8a3-da951b41869b" (UID: "55ddd71b-1fbf-4359-b8a3-da951b41869b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:13.959511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.959281    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-socket" pod "55ddd71b-1fbf-4359-b8a3-da951b41869b" (UID: "55ddd71b-1fbf-4359-b8a3-da951b41869b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:13.959511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.959363    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ddd71b-1fbf-4359-b8a3-da951b41869b-istiod-ca-cert" pod "55ddd71b-1fbf-4359-b8a3-da951b41869b" (UID: "55ddd71b-1fbf-4359-b8a3-da951b41869b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:13.959511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.959463    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:13.959511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.959475    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:13.959511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.959485    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:13.959511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.959494    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ddd71b-1fbf-4359-b8a3-da951b41869b-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:13.959511 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.959502    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:13.961254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.961227    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-envoy" pod "55ddd71b-1fbf-4359-b8a3-da951b41869b" (UID: "55ddd71b-1fbf-4359-b8a3-da951b41869b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:13.961365 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.961269    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-kube-api-access-mjhdb" pod "55ddd71b-1fbf-4359-b8a3-da951b41869b" (UID: "55ddd71b-1fbf-4359-b8a3-da951b41869b"). InnerVolumeSpecName "kube-api-access-mjhdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:13.961582 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.961527    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-podinfo" pod "55ddd71b-1fbf-4359-b8a3-da951b41869b" (UID: "55ddd71b-1fbf-4359-b8a3-da951b41869b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:49:13.961647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:13.961532    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-token" pod "55ddd71b-1fbf-4359-b8a3-da951b41869b" (UID: "55ddd71b-1fbf-4359-b8a3-da951b41869b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:14.060744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.060712    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:14.060744 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.060741    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjhdb\" (UniqueName: \"kubernetes.io/projected/55ddd71b-1fbf-4359-b8a3-da951b41869b-kube-api-access-mjhdb\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:14.060951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.060753    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:14.060951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.060761    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ddd71b-1fbf-4359-b8a3-da951b41869b-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:14.461915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.461819    4530 generic.go:356] "Generic (PLEG): container finished" podID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerID="f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac" exitCode=0
Oct 30 14:49:14.461915 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.461868    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"
Oct 30 14:49:14.462422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.461910    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" event={"ID":"55ddd71b-1fbf-4359-b8a3-da951b41869b","Type":"ContainerDied","Data":"f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac"}
Oct 30 14:49:14.462422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.461968    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr" event={"ID":"55ddd71b-1fbf-4359-b8a3-da951b41869b","Type":"ContainerDied","Data":"db37ae6dfcfb7ca21fe02ec584baa984b4d99517d1d7f5b2892b3f6322891635"}
Oct 30 14:49:14.462422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.461990    4530 scope.go:122] "RemoveContainer" containerID="f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac"
Oct 30 14:49:14.472680 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.472658    4530 scope.go:122] "RemoveContainer" containerID="f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa"
Oct 30 14:49:14.482675 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.482657    4530 scope.go:122] "RemoveContainer" containerID="f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac"
Oct 30 14:49:14.482975 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:14.482952    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac\": container with ID starting with f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac not found: ID does not exist" containerID="f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac"
Oct 30 14:49:14.483040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.482985    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac"} err="failed to get container status \"f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac\": rpc error: code = NotFound desc = could not find container \"f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac\": container with ID starting with f88fce0b6500a1d1baaa586c42e185e5098c41bf6fb0e302b8480e78a3eee3ac not found: ID does not exist"
Oct 30 14:49:14.483040 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.483004    4530 scope.go:122] "RemoveContainer" containerID="f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa"
Oct 30 14:49:14.483233 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:14.483214    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa\": container with ID starting with f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa not found: ID does not exist" containerID="f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa"
Oct 30 14:49:14.483277 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.483239    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa"} err="failed to get container status \"f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa\": rpc error: code = NotFound desc = could not find container \"f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa\": container with ID starting with f50b4819673303b9cc38231e3b6fb5fa62d62d8cf31894188ba97054ffe92eaa not found: ID does not exist"
Oct 30 14:49:14.525237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.525203    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"]
Oct 30 14:49:14.539123 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:14.539090    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3b95dd-openshift-default-75fccfd4c9-c7jlr"]
Oct 30 14:49:15.114602 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:15.114566    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3214e243-6b64-417d-8711-4b3cacb3c1bb" path="/var/lib/kubelet/pods/3214e243-6b64-417d-8711-4b3cacb3c1bb/volumes"
Oct 30 14:49:15.115045 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:15.115032    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" path="/var/lib/kubelet/pods/55ddd71b-1fbf-4359-b8a3-da951b41869b/volumes"
Oct 30 14:49:22.284008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:22.283962    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"]
Oct 30 14:49:22.284443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:22.284254    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy" containerID="cri-o://e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035" gracePeriod=30
Oct 30 14:49:25.211263 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:25.211224    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"]
Oct 30 14:49:27.544751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.544723    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:49:27.549588 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.549529    4530 generic.go:356] "Generic (PLEG): container finished" podID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerID="e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035" exitCode=0
Oct 30 14:49:27.549713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.549611    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"
Oct 30 14:49:27.549713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.549631    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" event={"ID":"daef6f1c-c355-4357-aa4c-f94ba2619dd9","Type":"ContainerDied","Data":"e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035"}
Oct 30 14:49:27.549713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.549691    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g" event={"ID":"daef6f1c-c355-4357-aa4c-f94ba2619dd9","Type":"ContainerDied","Data":"19e29dfcb19d83d6a7821d6d26398d3b400643eb8513e18dc7ca448d6a022059"}
Oct 30 14:49:27.549840 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.549715    4530 scope.go:122] "RemoveContainer" containerID="e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035"
Oct 30 14:49:27.559617 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.559598    4530 scope.go:122] "RemoveContainer" containerID="049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3"
Oct 30 14:49:27.568593 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.568574    4530 scope.go:122] "RemoveContainer" containerID="e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035"
Oct 30 14:49:27.568855 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:27.568835    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035\": container with ID starting with e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035 not found: ID does not exist" containerID="e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035"
Oct 30 14:49:27.568912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.568864    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035"} err="failed to get container status \"e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035\": rpc error: code = NotFound desc = could not find container \"e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035\": container with ID starting with e0eae26e13e79300089f49c8f957398e33eb498706d1d268e9022bc7f3132035 not found: ID does not exist"
Oct 30 14:49:27.568912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.568889    4530 scope.go:122] "RemoveContainer" containerID="049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3"
Oct 30 14:49:27.569130 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:27.569112    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3\": container with ID starting with 049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3 not found: ID does not exist" containerID="049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3"
Oct 30 14:49:27.569169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.569138    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3"} err="failed to get container status \"049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3\": rpc error: code = NotFound desc = could not find container \"049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3\": container with ID starting with 049f915571ad4101462f2b18c0b2531f5695fdf2719bb1d4d91956cf3aa1d5c3 not found: ID does not exist"
Oct 30 14:49:27.597910 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.597883    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-kube-api-access-xbxq7\" (UniqueName: \"kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-kube-api-access-xbxq7\") pod \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") "
Oct 30 14:49:27.598016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.597922    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-credential-socket\") pod \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") "
Oct 30 14:49:27.598016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.597956    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-envoy\") pod \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") "
Oct 30 14:49:27.598140 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.598026    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-token\" (UniqueName: \"kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-token\") pod \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") "
Oct 30 14:49:27.598140 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.598065    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-data\") pod \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") "
Oct 30 14:49:27.598246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.598155    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istiod-ca-cert\") pod \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") "
Oct 30 14:49:27.598246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.598158    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-credential-socket" pod "daef6f1c-c355-4357-aa4c-f94ba2619dd9" (UID: "daef6f1c-c355-4357-aa4c-f94ba2619dd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:27.598246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.598197    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-socket\") pod \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") "
Oct 30 14:49:27.598395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.598258    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-certs\") pod \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") "
Oct 30 14:49:27.598395 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.598301    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-podinfo\") pod \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\" (UID: \"daef6f1c-c355-4357-aa4c-f94ba2619dd9\") "
Oct 30 14:49:27.598488 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.598454    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-socket" pod "daef6f1c-c355-4357-aa4c-f94ba2619dd9" (UID: "daef6f1c-c355-4357-aa4c-f94ba2619dd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:27.598837 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.598790    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-certs" pod "daef6f1c-c355-4357-aa4c-f94ba2619dd9" (UID: "daef6f1c-c355-4357-aa4c-f94ba2619dd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:27.599115 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.598394    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-data" pod "daef6f1c-c355-4357-aa4c-f94ba2619dd9" (UID: "daef6f1c-c355-4357-aa4c-f94ba2619dd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:27.599236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.599210    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istiod-ca-cert" pod "daef6f1c-c355-4357-aa4c-f94ba2619dd9" (UID: "daef6f1c-c355-4357-aa4c-f94ba2619dd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:27.600676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.600646    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-kube-api-access-xbxq7" pod "daef6f1c-c355-4357-aa4c-f94ba2619dd9" (UID: "daef6f1c-c355-4357-aa4c-f94ba2619dd9"). InnerVolumeSpecName "kube-api-access-xbxq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:27.600820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.600790    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:27.600949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.600832    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:27.600949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.600849    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:27.600949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.600862    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:27.600949 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.600886    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:27.611462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.611437    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-token" pod "daef6f1c-c355-4357-aa4c-f94ba2619dd9" (UID: "daef6f1c-c355-4357-aa4c-f94ba2619dd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:27.611850 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.611825    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-podinfo" pod "daef6f1c-c355-4357-aa4c-f94ba2619dd9" (UID: "daef6f1c-c355-4357-aa4c-f94ba2619dd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:49:27.611970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.611823    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-envoy" pod "daef6f1c-c355-4357-aa4c-f94ba2619dd9" (UID: "daef6f1c-c355-4357-aa4c-f94ba2619dd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:27.701618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.701586    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:27.701618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.701613    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbxq7\" (UniqueName: \"kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-kube-api-access-xbxq7\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:27.701618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.701623    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:27.701856 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.701631    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daef6f1c-c355-4357-aa4c-f94ba2619dd9-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:27.902314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.902285    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"]
Oct 30 14:49:27.908454 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:27.908425    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-thj9g"]
Oct 30 14:49:28.075090 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:28.075044    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:49:29.114241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:29.114195    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" path="/var/lib/kubelet/pods/daef6f1c-c355-4357-aa4c-f94ba2619dd9/volumes"
Oct 30 14:49:30.212346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.212304    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk" podUID="61e81b9f-09ff-4a4f-bcef-75c443a1af7f" containerName="payload-pre-processing" containerID="cri-o://c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c" gracePeriod=25
Oct 30 14:49:30.485788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.485761    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"
Oct 30 14:49:30.531431 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.531398    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-plugins-config-volume\") pod \"61e81b9f-09ff-4a4f-bcef-75c443a1af7f\" (UID: \"61e81b9f-09ff-4a4f-bcef-75c443a1af7f\") "
Oct 30 14:49:30.531629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.531566    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-kube-api-access-znnzj\" (UniqueName: \"kubernetes.io/projected/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-kube-api-access-znnzj\") pod \"61e81b9f-09ff-4a4f-bcef-75c443a1af7f\" (UID: \"61e81b9f-09ff-4a4f-bcef-75c443a1af7f\") "
Oct 30 14:49:30.532291 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.532022    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-plugins-config-volume" pod "61e81b9f-09ff-4a4f-bcef-75c443a1af7f" (UID: "61e81b9f-09ff-4a4f-bcef-75c443a1af7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:30.532771 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.532717    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:30.534368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.534330    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-kube-api-access-znnzj" pod "61e81b9f-09ff-4a4f-bcef-75c443a1af7f" (UID: "61e81b9f-09ff-4a4f-bcef-75c443a1af7f"). InnerVolumeSpecName "kube-api-access-znnzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:30.573713 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.573681    4530 generic.go:356] "Generic (PLEG): container finished" podID="61e81b9f-09ff-4a4f-bcef-75c443a1af7f" containerID="c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c" exitCode=0
Oct 30 14:49:30.573834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.573733    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk" event={"ID":"61e81b9f-09ff-4a4f-bcef-75c443a1af7f","Type":"ContainerDied","Data":"c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c"}
Oct 30 14:49:30.573834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.573771    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk" event={"ID":"61e81b9f-09ff-4a4f-bcef-75c443a1af7f","Type":"ContainerDied","Data":"4c3844e73cb766efc1e76944a57e9509c67c0c832a8f681e48cdd25c05e6de2b"}
Oct 30 14:49:30.573834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.573790    4530 scope.go:122] "RemoveContainer" containerID="c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c"
Oct 30 14:49:30.573834 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.573735    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"
Oct 30 14:49:30.583800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.583784    4530 scope.go:122] "RemoveContainer" containerID="c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c"
Oct 30 14:49:30.584093 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:30.584074    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c\": container with ID starting with c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c not found: ID does not exist" containerID="c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c"
Oct 30 14:49:30.584165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.584102    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c"} err="failed to get container status \"c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c\": rpc error: code = NotFound desc = could not find container \"c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c\": container with ID starting with c1998d4a593e8affbcb444238a374c431def9c292066422e08258c93922c362c not found: ID does not exist"
Oct 30 14:49:30.633718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.633684    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znnzj\" (UniqueName: \"kubernetes.io/projected/61e81b9f-09ff-4a4f-bcef-75c443a1af7f-kube-api-access-znnzj\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:30.636723 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.636700    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"]
Oct 30 14:49:30.639336 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.639315    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f1f206-dbc6955ff-k76fk"]
Oct 30 14:49:30.907958 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.907924    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"]
Oct 30 14:49:30.908159 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:30.908133    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5" podUID="ad23228d-09a6-4eb9-a818-746f037fa303" containerName="maas-api" containerID="cri-o://a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e" gracePeriod=30
Oct 30 14:49:31.115373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.115334    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e81b9f-09ff-4a4f-bcef-75c443a1af7f" path="/var/lib/kubelet/pods/61e81b9f-09ff-4a4f-bcef-75c443a1af7f/volumes"
Oct 30 14:49:31.190813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.190779    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:49:31.239241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.239209    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad23228d-09a6-4eb9-a818-746f037fa303-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad23228d-09a6-4eb9-a818-746f037fa303-openshift-service-ca\") pod \"ad23228d-09a6-4eb9-a818-746f037fa303\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") "
Oct 30 14:49:31.239747 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.239306    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-maas-api-tls\") pod \"ad23228d-09a6-4eb9-a818-746f037fa303\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") "
Oct 30 14:49:31.239747 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.239406    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad23228d-09a6-4eb9-a818-746f037fa303-kube-api-access-lb924\" (UniqueName: \"kubernetes.io/projected/ad23228d-09a6-4eb9-a818-746f037fa303-kube-api-access-lb924\") pod \"ad23228d-09a6-4eb9-a818-746f037fa303\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") "
Oct 30 14:49:31.239747 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.239676    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-metrics-certs\") pod \"ad23228d-09a6-4eb9-a818-746f037fa303\" (UID: \"ad23228d-09a6-4eb9-a818-746f037fa303\") "
Oct 30 14:49:31.242234 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.242202    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-maas-api-tls" pod "ad23228d-09a6-4eb9-a818-746f037fa303" (UID: "ad23228d-09a6-4eb9-a818-746f037fa303"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:49:31.242696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.242671    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad23228d-09a6-4eb9-a818-746f037fa303-kube-api-access-lb924" pod "ad23228d-09a6-4eb9-a818-746f037fa303" (UID: "ad23228d-09a6-4eb9-a818-746f037fa303"). InnerVolumeSpecName "kube-api-access-lb924". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:31.243666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.243105    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad23228d-09a6-4eb9-a818-746f037fa303-openshift-service-ca" pod "ad23228d-09a6-4eb9-a818-746f037fa303" (UID: "ad23228d-09a6-4eb9-a818-746f037fa303"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:31.243666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.243641    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-metrics-certs" pod "ad23228d-09a6-4eb9-a818-746f037fa303" (UID: "ad23228d-09a6-4eb9-a818-746f037fa303"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:49:31.340902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.340855    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:31.340902 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.340901    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb924\" (UniqueName: \"kubernetes.io/projected/ad23228d-09a6-4eb9-a818-746f037fa303-kube-api-access-lb924\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:31.341188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.340916    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad23228d-09a6-4eb9-a818-746f037fa303-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:31.341188 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.340932    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad23228d-09a6-4eb9-a818-746f037fa303-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:31.343014 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.342980    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"]
Oct 30 14:49:31.343235 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.343207    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq" podUID="781c45e8-e9fc-4928-b487-3371e529e361" containerName="maas-api" containerID="cri-o://b829244fc49b35da6a256ad2a0d98aca6699e7eb200be2df695abb47425461e9" gracePeriod=30
Oct 30 14:49:31.609914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.608919    4530 generic.go:356] "Generic (PLEG): container finished" podID="781c45e8-e9fc-4928-b487-3371e529e361" containerID="b829244fc49b35da6a256ad2a0d98aca6699e7eb200be2df695abb47425461e9" exitCode=0
Oct 30 14:49:31.609914 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.609073    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq" event={"ID":"781c45e8-e9fc-4928-b487-3371e529e361","Type":"ContainerDied","Data":"b829244fc49b35da6a256ad2a0d98aca6699e7eb200be2df695abb47425461e9"}
Oct 30 14:49:31.632049 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.632008    4530 generic.go:356] "Generic (PLEG): container finished" podID="ad23228d-09a6-4eb9-a818-746f037fa303" containerID="a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e" exitCode=0
Oct 30 14:49:31.643400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.632418    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5" event={"ID":"ad23228d-09a6-4eb9-a818-746f037fa303","Type":"ContainerDied","Data":"a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e"}
Oct 30 14:49:31.643661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.638023    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"
Oct 30 14:49:31.643661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.643647    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5" event={"ID":"ad23228d-09a6-4eb9-a818-746f037fa303","Type":"ContainerDied","Data":"f640347246156b10a6aad4827516ecf3a22f99ab6b47df9e07f944fd694a1e0c"}
Oct 30 14:49:31.643867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.643724    4530 scope.go:122] "RemoveContainer" containerID="a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e"
Oct 30 14:49:31.649134 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.649114    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:49:31.656630 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.656613    4530 scope.go:122] "RemoveContainer" containerID="a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e"
Oct 30 14:49:31.656934 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:31.656914    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e\": container with ID starting with a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e not found: ID does not exist" containerID="a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e"
Oct 30 14:49:31.657001 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.656943    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e"} err="failed to get container status \"a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e\": rpc error: code = NotFound desc = could not find container \"a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e\": container with ID starting with a11fc0268c6b99f15054d54074c79631fa69030ebcb79354260493cec320586e not found: ID does not exist"
Oct 30 14:49:31.708150 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.708085    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"]
Oct 30 14:49:31.722425 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.722389    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f1f206-675b685b47-qg4m5"]
Oct 30 14:49:31.745527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.745494    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-metrics-certs\" (UniqueName: \"kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-metrics-certs\") pod \"781c45e8-e9fc-4928-b487-3371e529e361\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") "
Oct 30 14:49:31.745721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.745578    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/781c45e8-e9fc-4928-b487-3371e529e361-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/781c45e8-e9fc-4928-b487-3371e529e361-openshift-service-ca\") pod \"781c45e8-e9fc-4928-b487-3371e529e361\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") "
Oct 30 14:49:31.745721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.745643    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781c45e8-e9fc-4928-b487-3371e529e361-kube-api-access-d94sf\" (UniqueName: \"kubernetes.io/projected/781c45e8-e9fc-4928-b487-3371e529e361-kube-api-access-d94sf\") pod \"781c45e8-e9fc-4928-b487-3371e529e361\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") "
Oct 30 14:49:31.745838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.745823    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-maas-api-tls\") pod \"781c45e8-e9fc-4928-b487-3371e529e361\" (UID: \"781c45e8-e9fc-4928-b487-3371e529e361\") "
Oct 30 14:49:31.746048 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.746015    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781c45e8-e9fc-4928-b487-3371e529e361-openshift-service-ca" pod "781c45e8-e9fc-4928-b487-3371e529e361" (UID: "781c45e8-e9fc-4928-b487-3371e529e361"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:31.746342 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.746205    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/781c45e8-e9fc-4928-b487-3371e529e361-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:31.751647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.748356    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781c45e8-e9fc-4928-b487-3371e529e361-kube-api-access-d94sf" pod "781c45e8-e9fc-4928-b487-3371e529e361" (UID: "781c45e8-e9fc-4928-b487-3371e529e361"). InnerVolumeSpecName "kube-api-access-d94sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:31.751647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.748363    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-metrics-certs" pod "781c45e8-e9fc-4928-b487-3371e529e361" (UID: "781c45e8-e9fc-4928-b487-3371e529e361"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:49:31.760396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.760357    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-maas-api-tls" pod "781c45e8-e9fc-4928-b487-3371e529e361" (UID: "781c45e8-e9fc-4928-b487-3371e529e361"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:49:31.846952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.846914    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:31.846952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.846943    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781c45e8-e9fc-4928-b487-3371e529e361-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:31.846952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:31.846952    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d94sf\" (UniqueName: \"kubernetes.io/projected/781c45e8-e9fc-4928-b487-3371e529e361-kube-api-access-d94sf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:32.653874 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:32.653837    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq" event={"ID":"781c45e8-e9fc-4928-b487-3371e529e361","Type":"ContainerDied","Data":"b619b6ad7456e6145e8b792c7947f84570e62037ba18378f25fc966f42662fa7"}
Oct 30 14:49:32.654325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:32.653879    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"
Oct 30 14:49:32.654325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:32.653906    4530 scope.go:122] "RemoveContainer" containerID="b829244fc49b35da6a256ad2a0d98aca6699e7eb200be2df695abb47425461e9"
Oct 30 14:49:32.714447 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:32.714413    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"]
Oct 30 14:49:32.728846 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:32.728810    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-75443e-857c8cf9cf-8jxkq"]
Oct 30 14:49:33.113720 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:33.113685    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="781c45e8-e9fc-4928-b487-3371e529e361" path="/var/lib/kubelet/pods/781c45e8-e9fc-4928-b487-3371e529e361/volumes"
Oct 30 14:49:33.114140 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:33.114127    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad23228d-09a6-4eb9-a818-746f037fa303" path="/var/lib/kubelet/pods/ad23228d-09a6-4eb9-a818-746f037fa303/volumes"
Oct 30 14:49:41.344100 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.344008    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"]
Oct 30 14:49:41.344702 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.344262    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy" containerID="cri-o://cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc" gracePeriod=30
Oct 30 14:49:41.367812 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:41.367776    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:49:41.874092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874057    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"]
Oct 30 14:49:41.874507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874494    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy"
Oct 30 14:49:41.874573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874508    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy"
Oct 30 14:49:41.874573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874524    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy"
Oct 30 14:49:41.874573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874529    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy"
Oct 30 14:49:41.874573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874551    4530 cpu_manager.go:414] "containerMap: removing container" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy"
Oct 30 14:49:41.874573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874558    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy"
Oct 30 14:49:41.874573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874566    4530 cpu_manager.go:414] "containerMap: removing container" podUID="3214e243-6b64-417d-8711-4b3cacb3c1bb" containerName="maas-api"
Oct 30 14:49:41.874573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874571    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3214e243-6b64-417d-8711-4b3cacb3c1bb" containerName="maas-api"
Oct 30 14:49:41.874573 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874578    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8240e208-5f61-45a4-b0e2-e6298ce7ab8e" containerName="payload-processing"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874583    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8240e208-5f61-45a4-b0e2-e6298ce7ab8e" containerName="payload-processing"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874591    4530 cpu_manager.go:414] "containerMap: removing container" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874595    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874600    4530 cpu_manager.go:414] "containerMap: removing container" podUID="940bb1b7-7692-4676-8b62-5f283e2b0811" containerName="maas-api"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874605    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940bb1b7-7692-4676-8b62-5f283e2b0811" containerName="maas-api"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874610    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c0852550-3a42-4c43-9c97-c73ac997c4d7" containerName="payload-pre-processing"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874615    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0852550-3a42-4c43-9c97-c73ac997c4d7" containerName="payload-pre-processing"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874622    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874626    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874633    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874637    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874644    4530 cpu_manager.go:414] "containerMap: removing container" podUID="c1ab0398-256e-4d59-814a-bde23499d65d" containerName="payload-processing"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874649    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ab0398-256e-4d59-814a-bde23499d65d" containerName="payload-processing"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874702    4530 cpu_manager.go:414] "containerMap: removing container" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874708    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874715    4530 cpu_manager.go:414] "containerMap: removing container" podUID="61e81b9f-09ff-4a4f-bcef-75c443a1af7f" containerName="payload-pre-processing"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874720    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e81b9f-09ff-4a4f-bcef-75c443a1af7f" containerName="payload-pre-processing"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874727    4530 cpu_manager.go:414] "containerMap: removing container" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874731    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874742    4530 cpu_manager.go:414] "containerMap: removing container" podUID="85899396-118d-4899-a445-2077f6778fc0" containerName="maas-api"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874747    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85899396-118d-4899-a445-2077f6778fc0" containerName="maas-api"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874753    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874758    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874764    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874768    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874779    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7aa3d073-f26b-49a3-b32e-25dea19ddf12" containerName="payload-processing"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874783    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa3d073-f26b-49a3-b32e-25dea19ddf12" containerName="payload-processing"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874793    4530 cpu_manager.go:414] "containerMap: removing container" podUID="52ec6eeb-78f9-4100-8390-c9688a8f94dc" containerName="payload-pre-processing"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874797    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ec6eeb-78f9-4100-8390-c9688a8f94dc" containerName="payload-pre-processing"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874806    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d93cd08a-fef5-441b-9898-799dba609917" containerName="maas-api"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874811    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93cd08a-fef5-441b-9898-799dba609917" containerName="maas-api"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874819    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874824    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874831    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" containerName="maas-api"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874836    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" containerName="maas-api"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874844    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" containerName="maas-api"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874849    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" containerName="maas-api"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874855    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9" containerName="maas-api"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874859    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9" containerName="maas-api"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874866    4530 cpu_manager.go:414] "containerMap: removing container" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874871    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874879    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874883    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874890    4530 cpu_manager.go:414] "containerMap: removing container" podUID="34398947-0d72-4b23-b8f8-e61dd6956a1a" containerName="istio-proxy"
Oct 30 14:49:41.874877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874894    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34398947-0d72-4b23-b8f8-e61dd6956a1a" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874901    4530 cpu_manager.go:414] "containerMap: removing container" podUID="d4ce80eb-7159-47c6-a5b5-cf94871d1e46" containerName="payload-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874905    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ce80eb-7159-47c6-a5b5-cf94871d1e46" containerName="payload-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874916    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9223ac67-5eec-4e34-a626-d585e44e5444" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874921    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9223ac67-5eec-4e34-a626-d585e44e5444" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874926    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f59a9fe9-5a43-4fb8-8446-6b13af4457e9" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874931    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59a9fe9-5a43-4fb8-8446-6b13af4457e9" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874938    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874943    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874950    4530 cpu_manager.go:414] "containerMap: removing container" podUID="781c45e8-e9fc-4928-b487-3371e529e361" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874954    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="781c45e8-e9fc-4928-b487-3371e529e361" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874961    4530 cpu_manager.go:414] "containerMap: removing container" podUID="a112e79e-e7e5-42c6-a74a-6c63e15fdc1e" containerName="payload-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874965    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a112e79e-e7e5-42c6-a74a-6c63e15fdc1e" containerName="payload-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874971    4530 cpu_manager.go:414] "containerMap: removing container" podUID="06ef73ec-3562-49a8-a436-515403188436" containerName="payload-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874975    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ef73ec-3562-49a8-a436-515403188436" containerName="payload-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874993    4530 cpu_manager.go:414] "containerMap: removing container" podUID="8628b21b-fb85-4411-b29f-9b23ef9c05a8" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.874998    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8628b21b-fb85-4411-b29f-9b23ef9c05a8" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875005    4530 cpu_manager.go:414] "containerMap: removing container" podUID="ad23228d-09a6-4eb9-a818-746f037fa303" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875009    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad23228d-09a6-4eb9-a818-746f037fa303" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875015    4530 cpu_manager.go:414] "containerMap: removing container" podUID="05d6ddc5-2dd7-4148-be66-b0b0ae909d37" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875020    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d6ddc5-2dd7-4148-be66-b0b0ae909d37" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875099    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e81b9f-09ff-4a4f-bcef-75c443a1af7f" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875109    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875115    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="85899396-118d-4899-a445-2077f6778fc0" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875121    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875126    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875134    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa3d073-f26b-49a3-b32e-25dea19ddf12" containerName="payload-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875141    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ec6eeb-78f9-4100-8390-c9688a8f94dc" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875147    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93cd08a-fef5-441b-9898-799dba609917" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875154    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875159    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875165    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab4de72-e77d-46aa-afb2-f6cde66ef7b9" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875170    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a2193e-90af-42cd-8af9-6cdcdaeeb7e9" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875176    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875180    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1cb57b5-00f4-4ef0-ad95-c9e99a20f32f" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875185    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="34398947-0d72-4b23-b8f8-e61dd6956a1a" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875189    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ce80eb-7159-47c6-a5b5-cf94871d1e46" containerName="payload-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875194    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9223ac67-5eec-4e34-a626-d585e44e5444" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875200    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59a9fe9-5a43-4fb8-8446-6b13af4457e9" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875205    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a82d8a8-0574-4500-80f1-6fdd38b4c8b1" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875210    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="781c45e8-e9fc-4928-b487-3371e529e361" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875217    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="a112e79e-e7e5-42c6-a74a-6c63e15fdc1e" containerName="payload-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875223    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ef73ec-3562-49a8-a436-515403188436" containerName="payload-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875229    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8628b21b-fb85-4411-b29f-9b23ef9c05a8" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875234    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad23228d-09a6-4eb9-a818-746f037fa303" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875239    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875244    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d6ddc5-2dd7-4148-be66-b0b0ae909d37" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875249    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875254    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57850de-5a13-4f64-9e13-004cbd6eb898" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875259    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="3214e243-6b64-417d-8711-4b3cacb3c1bb" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875265    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="8240e208-5f61-45a4-b0e2-e6298ce7ab8e" containerName="payload-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875270    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="276cdf23-88b0-4fc2-a20c-c5c4c27db30f" containerName="main"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875275    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="940bb1b7-7692-4676-8b62-5f283e2b0811" containerName="maas-api"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875279    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0852550-3a42-4c43-9c97-c73ac997c4d7" containerName="payload-pre-processing"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875284    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy"
Oct 30 14:49:41.876357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.875288    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ab0398-256e-4d59-814a-bde23499d65d" containerName="payload-processing"
Oct 30 14:49:41.880109 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.880088    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:41.897513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.897480    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"]
Oct 30 14:49:41.907230 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.907203    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:49:41.907350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.907244    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:49:41.956649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.955360    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7p8w\" (UniqueName: \"kubernetes.io/projected/49075769-3848-432c-8af5-7d1a4b07b13f-kube-api-access-j7p8w\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht\" (UID: \"49075769-3848-432c-8af5-7d1a4b07b13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:41.956649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.955418    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht\" (UID: \"49075769-3848-432c-8af5-7d1a4b07b13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:41.956649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:41.955572    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht\" (UID: \"49075769-3848-432c-8af5-7d1a4b07b13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:42.056205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:42.056121    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7p8w\" (UniqueName: \"kubernetes.io/projected/49075769-3848-432c-8af5-7d1a4b07b13f-kube-api-access-j7p8w\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht\" (UID: \"49075769-3848-432c-8af5-7d1a4b07b13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:42.056205 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:42.056177    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht\" (UID: \"49075769-3848-432c-8af5-7d1a4b07b13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:42.056471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:42.056345    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht\" (UID: \"49075769-3848-432c-8af5-7d1a4b07b13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:42.057138 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:42.057106    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht\" (UID: \"49075769-3848-432c-8af5-7d1a4b07b13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:42.067613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:42.059215    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht\" (UID: \"49075769-3848-432c-8af5-7d1a4b07b13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:42.084031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:42.083992    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7p8w\" (UniqueName: \"kubernetes.io/projected/49075769-3848-432c-8af5-7d1a4b07b13f-kube-api-access-j7p8w\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht\" (UID: \"49075769-3848-432c-8af5-7d1a4b07b13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:42.195086 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:42.195005    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:42.502189 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:49:42.502151    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49075769_3848_432c_8af5_7d1a4b07b13f.slice/crio-fc141849649a6ad33f4b73a8c564e8c0ca3220bf80810060780b416ee3774383 WatchSource:0}: Error finding container fc141849649a6ad33f4b73a8c564e8c0ca3220bf80810060780b416ee3774383: Status 404 returned error can't find the container with id fc141849649a6ad33f4b73a8c564e8c0ca3220bf80810060780b416ee3774383
Oct 30 14:49:42.502696 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:42.502669    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"]
Oct 30 14:49:42.717787 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:42.717749    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht" event={"ID":"49075769-3848-432c-8af5-7d1a4b07b13f","Type":"ContainerStarted","Data":"fc141849649a6ad33f4b73a8c564e8c0ca3220bf80810060780b416ee3774383"}
Oct 30 14:49:43.726207 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:43.726109    4530 generic.go:356] "Generic (PLEG): container finished" podID="49075769-3848-432c-8af5-7d1a4b07b13f" containerID="1683434a035bd3122fd53a64b781a2d22f554e43dad0a11a33c137a6c8fa65a3" exitCode=0
Oct 30 14:49:43.726644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:43.726195    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht" event={"ID":"49075769-3848-432c-8af5-7d1a4b07b13f","Type":"ContainerDied","Data":"1683434a035bd3122fd53a64b781a2d22f554e43dad0a11a33c137a6c8fa65a3"}
Oct 30 14:49:44.877912 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:44.877876    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:44.987420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:44.987324    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-xks-ca\") pod \"49075769-3848-432c-8af5-7d1a4b07b13f\" (UID: \"49075769-3848-432c-8af5-7d1a4b07b13f\") "
Oct 30 14:49:44.987420 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:44.987371    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49075769-3848-432c-8af5-7d1a4b07b13f-kube-api-access-j7p8w\" (UniqueName: \"kubernetes.io/projected/49075769-3848-432c-8af5-7d1a4b07b13f-kube-api-access-j7p8w\") pod \"49075769-3848-432c-8af5-7d1a4b07b13f\" (UID: \"49075769-3848-432c-8af5-7d1a4b07b13f\") "
Oct 30 14:49:44.987677 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:44.987452    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-openshift-ca\") pod \"49075769-3848-432c-8af5-7d1a4b07b13f\" (UID: \"49075769-3848-432c-8af5-7d1a4b07b13f\") "
Oct 30 14:49:44.987891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:44.987868    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-openshift-ca" pod "49075769-3848-432c-8af5-7d1a4b07b13f" (UID: "49075769-3848-432c-8af5-7d1a4b07b13f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:44.989496 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:44.989468    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-xks-ca" pod "49075769-3848-432c-8af5-7d1a4b07b13f" (UID: "49075769-3848-432c-8af5-7d1a4b07b13f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:49:44.989638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:44.989614    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49075769-3848-432c-8af5-7d1a4b07b13f-kube-api-access-j7p8w" pod "49075769-3848-432c-8af5-7d1a4b07b13f" (UID: "49075769-3848-432c-8af5-7d1a4b07b13f"). InnerVolumeSpecName "kube-api-access-j7p8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:45.089193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:45.089148    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:45.089193 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:45.089188    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49075769-3848-432c-8af5-7d1a4b07b13f-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:45.089421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:45.089203    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7p8w\" (UniqueName: \"kubernetes.io/projected/49075769-3848-432c-8af5-7d1a4b07b13f-kube-api-access-j7p8w\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:45.546417 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:45.546380    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"]
Oct 30 14:49:45.579628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:45.579591    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"]
Oct 30 14:49:45.740597 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:45.740535    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht" event={"ID":"49075769-3848-432c-8af5-7d1a4b07b13f","Type":"ContainerDied","Data":"fc141849649a6ad33f4b73a8c564e8c0ca3220bf80810060780b416ee3774383"}
Oct 30 14:49:45.740799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:45.740623    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc141849649a6ad33f4b73a8c564e8c0ca3220bf80810060780b416ee3774383"
Oct 30 14:49:45.740799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:45.740573    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"
Oct 30 14:49:46.612762 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.612702    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:49:46.706556 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.706517    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-credential-socket\") pod \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") "
Oct 30 14:49:46.706759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.706592    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istiod-ca-cert\") pod \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") "
Oct 30 14:49:46.706759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.706631    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-data\") pod \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") "
Oct 30 14:49:46.706759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.706668    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-socket\") pod \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") "
Oct 30 14:49:46.706759 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.706756    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-certs\") pod \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") "
Oct 30 14:49:46.707032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.706786    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-envoy\") pod \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") "
Oct 30 14:49:46.707032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.706816    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-kube-api-access-fw29m\" (UniqueName: \"kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-kube-api-access-fw29m\") pod \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") "
Oct 30 14:49:46.707032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.706815    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-credential-socket" pod "b7bf62ef-287a-4ede-95e9-a0c8d8354141" (UID: "b7bf62ef-287a-4ede-95e9-a0c8d8354141"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:46.707032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.706857    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-data" pod "b7bf62ef-287a-4ede-95e9-a0c8d8354141" (UID: "b7bf62ef-287a-4ede-95e9-a0c8d8354141"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:46.707032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.706861    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-podinfo\") pod \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") "
Oct 30 14:49:46.707032 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.706907    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-token\") pod \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\" (UID: \"b7bf62ef-287a-4ede-95e9-a0c8d8354141\") "
Oct 30 14:49:46.707340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.707087    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-certs" pod "b7bf62ef-287a-4ede-95e9-a0c8d8354141" (UID: "b7bf62ef-287a-4ede-95e9-a0c8d8354141"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:46.707340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.707095    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istiod-ca-cert" pod "b7bf62ef-287a-4ede-95e9-a0c8d8354141" (UID: "b7bf62ef-287a-4ede-95e9-a0c8d8354141"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:46.707455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.707333    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-socket" pod "b7bf62ef-287a-4ede-95e9-a0c8d8354141" (UID: "b7bf62ef-287a-4ede-95e9-a0c8d8354141"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:46.707570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.707528    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:46.707667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.707577    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:46.707667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.707594    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:46.707667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.707608    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:46.707667 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.707620    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:46.709081 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.709049    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-podinfo" pod "b7bf62ef-287a-4ede-95e9-a0c8d8354141" (UID: "b7bf62ef-287a-4ede-95e9-a0c8d8354141"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:49:46.709479 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.709448    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-envoy" pod "b7bf62ef-287a-4ede-95e9-a0c8d8354141" (UID: "b7bf62ef-287a-4ede-95e9-a0c8d8354141"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:46.709628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.709605    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-kube-api-access-fw29m" pod "b7bf62ef-287a-4ede-95e9-a0c8d8354141" (UID: "b7bf62ef-287a-4ede-95e9-a0c8d8354141"). InnerVolumeSpecName "kube-api-access-fw29m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:46.709694 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.709641    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-token" pod "b7bf62ef-287a-4ede-95e9-a0c8d8354141" (UID: "b7bf62ef-287a-4ede-95e9-a0c8d8354141"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:46.748013 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.747984    4530 generic.go:356] "Generic (PLEG): container finished" podID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerID="cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc" exitCode=0
Oct 30 14:49:46.748154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.748031    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"
Oct 30 14:49:46.748154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.748060    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" event={"ID":"b7bf62ef-287a-4ede-95e9-a0c8d8354141","Type":"ContainerDied","Data":"cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc"}
Oct 30 14:49:46.748154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.748114    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2" event={"ID":"b7bf62ef-287a-4ede-95e9-a0c8d8354141","Type":"ContainerDied","Data":"9826a92c562fd48e150b265cd6dde355dbfc1611e43ac4237bad02a5917fbecb"}
Oct 30 14:49:46.748154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.748136    4530 scope.go:122] "RemoveContainer" containerID="cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc"
Oct 30 14:49:46.758378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.758361    4530 scope.go:122] "RemoveContainer" containerID="b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167"
Oct 30 14:49:46.767218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.767201    4530 scope.go:122] "RemoveContainer" containerID="cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc"
Oct 30 14:49:46.767486 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:46.767465    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc\": container with ID starting with cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc not found: ID does not exist" containerID="cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc"
Oct 30 14:49:46.767599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.767498    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc"} err="failed to get container status \"cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc\": rpc error: code = NotFound desc = could not find container \"cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc\": container with ID starting with cd14d9301be327962129f1b6fe292c4f731c52bbf5d496c533cdfad0e5408cfc not found: ID does not exist"
Oct 30 14:49:46.767599 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.767522    4530 scope.go:122] "RemoveContainer" containerID="b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167"
Oct 30 14:49:46.767911 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:46.767883    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167\": container with ID starting with b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167 not found: ID does not exist" containerID="b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167"
Oct 30 14:49:46.767963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.767910    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167"} err="failed to get container status \"b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167\": rpc error: code = NotFound desc = could not find container \"b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167\": container with ID starting with b8907883b82193bc862d054ff4370aada3b45bd5c3662027c868ef7cc0c75167 not found: ID does not exist"
Oct 30 14:49:46.808409 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.808380    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:46.808535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.808411    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw29m\" (UniqueName: \"kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-kube-api-access-fw29m\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:46.808535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.808426    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:46.808535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.808436    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf62ef-287a-4ede-95e9-a0c8d8354141-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:46.811687 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.811660    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"]
Oct 30 14:49:46.819536 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:46.819513    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-24sv2"]
Oct 30 14:49:47.089809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:47.089768    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"]
Oct 30 14:49:47.090048 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:47.090023    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerName="istio-proxy" containerID="cri-o://79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab" gracePeriod=30
Oct 30 14:49:47.113711 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:47.113686    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" path="/var/lib/kubelet/pods/b7bf62ef-287a-4ede-95e9-a0c8d8354141/volumes"
Oct 30 14:49:48.947018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:48.946927    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:49:51.254981 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:51.254948    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"]
Oct 30 14:49:52.353977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.353949    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:49:52.470228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.470122    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-kube-api-access-jrzdn\" (UniqueName: \"kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-kube-api-access-jrzdn\") pod \"7ac20408-a58d-4bc9-8526-b86fe83d9020\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") "
Oct 30 14:49:52.470228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.470208    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-token\" (UniqueName: \"kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-token\") pod \"7ac20408-a58d-4bc9-8526-b86fe83d9020\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") "
Oct 30 14:49:52.470636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.470259    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac20408-a58d-4bc9-8526-b86fe83d9020-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac20408-a58d-4bc9-8526-b86fe83d9020-istiod-ca-cert\") pod \"7ac20408-a58d-4bc9-8526-b86fe83d9020\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") "
Oct 30 14:49:52.470636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.470295    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-certs\") pod \"7ac20408-a58d-4bc9-8526-b86fe83d9020\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") "
Oct 30 14:49:52.470636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.470340    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-credential-socket\") pod \"7ac20408-a58d-4bc9-8526-b86fe83d9020\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") "
Oct 30 14:49:52.470636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.470426    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-envoy\") pod \"7ac20408-a58d-4bc9-8526-b86fe83d9020\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") "
Oct 30 14:49:52.470636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.470483    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-data\") pod \"7ac20408-a58d-4bc9-8526-b86fe83d9020\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") "
Oct 30 14:49:52.470636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.470519    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-socket\") pod \"7ac20408-a58d-4bc9-8526-b86fe83d9020\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") "
Oct 30 14:49:52.470984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.470689    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-podinfo\") pod \"7ac20408-a58d-4bc9-8526-b86fe83d9020\" (UID: \"7ac20408-a58d-4bc9-8526-b86fe83d9020\") "
Oct 30 14:49:52.470984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.470807    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-certs" pod "7ac20408-a58d-4bc9-8526-b86fe83d9020" (UID: "7ac20408-a58d-4bc9-8526-b86fe83d9020"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:52.471161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.471125    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:52.473742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.473608    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-token" pod "7ac20408-a58d-4bc9-8526-b86fe83d9020" (UID: "7ac20408-a58d-4bc9-8526-b86fe83d9020"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:52.473742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.473669    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-kube-api-access-jrzdn" pod "7ac20408-a58d-4bc9-8526-b86fe83d9020" (UID: "7ac20408-a58d-4bc9-8526-b86fe83d9020"). InnerVolumeSpecName "kube-api-access-jrzdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:49:52.474113 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.473950    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac20408-a58d-4bc9-8526-b86fe83d9020-istiod-ca-cert" pod "7ac20408-a58d-4bc9-8526-b86fe83d9020" (UID: "7ac20408-a58d-4bc9-8526-b86fe83d9020"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:49:52.474274 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.474214    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-credential-socket" pod "7ac20408-a58d-4bc9-8526-b86fe83d9020" (UID: "7ac20408-a58d-4bc9-8526-b86fe83d9020"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:52.474402 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.474387    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-data" pod "7ac20408-a58d-4bc9-8526-b86fe83d9020" (UID: "7ac20408-a58d-4bc9-8526-b86fe83d9020"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:52.474619 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.474583    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-socket" pod "7ac20408-a58d-4bc9-8526-b86fe83d9020" (UID: "7ac20408-a58d-4bc9-8526-b86fe83d9020"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:52.474777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.474757    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-podinfo" pod "7ac20408-a58d-4bc9-8526-b86fe83d9020" (UID: "7ac20408-a58d-4bc9-8526-b86fe83d9020"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:49:52.476195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.476164    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-envoy" pod "7ac20408-a58d-4bc9-8526-b86fe83d9020" (UID: "7ac20408-a58d-4bc9-8526-b86fe83d9020"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:49:52.577019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.576854    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:52.577019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.576904    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac20408-a58d-4bc9-8526-b86fe83d9020-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:52.577019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.576918    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:52.577019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.576935    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:52.577019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.576949    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:52.577019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.576962    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac20408-a58d-4bc9-8526-b86fe83d9020-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:52.577019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.576976    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac20408-a58d-4bc9-8526-b86fe83d9020-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:52.577019 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.576989    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrzdn\" (UniqueName: \"kubernetes.io/projected/7ac20408-a58d-4bc9-8526-b86fe83d9020-kube-api-access-jrzdn\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:49:52.788991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.788889    4530 generic.go:356] "Generic (PLEG): container finished" podID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerID="79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab" exitCode=0
Oct 30 14:49:52.788991 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.788951    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"
Oct 30 14:49:52.789246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.788974    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" event={"ID":"7ac20408-a58d-4bc9-8526-b86fe83d9020","Type":"ContainerDied","Data":"79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab"}
Oct 30 14:49:52.789246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.789046    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq" event={"ID":"7ac20408-a58d-4bc9-8526-b86fe83d9020","Type":"ContainerDied","Data":"97c8b36ca8f9aac798cd8ff90b145f3098e84d25a91edec2565135bfa455c968"}
Oct 30 14:49:52.789246 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.789075    4530 scope.go:122] "RemoveContainer" containerID="79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab"
Oct 30 14:49:52.800608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.800491    4530 scope.go:122] "RemoveContainer" containerID="3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0"
Oct 30 14:49:52.813058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.810936    4530 scope.go:122] "RemoveContainer" containerID="79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab"
Oct 30 14:49:52.813058 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:52.812876    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab\": container with ID starting with 79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab not found: ID does not exist" containerID="79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab"
Oct 30 14:49:52.813058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.812925    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab"} err="failed to get container status \"79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab\": rpc error: code = NotFound desc = could not find container \"79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab\": container with ID starting with 79b771057733a16caeeb46e220b4f9bee65b683f60344f15d1e8a567a07d2cab not found: ID does not exist"
Oct 30 14:49:52.813058 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.812950    4530 scope.go:122] "RemoveContainer" containerID="3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0"
Oct 30 14:49:52.813508 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:49:52.813481    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0\": container with ID starting with 3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0 not found: ID does not exist" containerID="3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0"
Oct 30 14:49:52.813724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.813513    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0"} err="failed to get container status \"3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0\": rpc error: code = NotFound desc = could not find container \"3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0\": container with ID starting with 3079f62e45ca4787f3062539e1e6cffe1d5ab55a5d1b38cd0701f64d8a3fabe0 not found: ID does not exist"
Oct 30 14:49:52.851157 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.851133    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"]
Oct 30 14:49:52.865586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:52.865551    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5704d6-openshift-default-84b89c8cb9-gwzxq"]
Oct 30 14:49:53.114174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:53.114140    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" path="/var/lib/kubelet/pods/7ac20408-a58d-4bc9-8526-b86fe83d9020/volumes"
Oct 30 14:49:56.255685 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:56.255641    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj" podUID="f4ba7a80-b30b-40b3-a209-674a54c6f096" containerName="payload-pre-processing" containerID="cri-o://9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798" gracePeriod=25
Oct 30 14:49:56.858039 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:56.857999    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"]
Oct 30 14:49:56.858282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:56.858256    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy" containerID="cri-o://061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b" gracePeriod=30
Oct 30 14:49:58.130332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.130297    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"]
Oct 30 14:49:58.130967 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.130944    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy"
Oct 30 14:49:58.131063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.130969    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy"
Oct 30 14:49:58.131063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.130986    4530 cpu_manager.go:414] "containerMap: removing container" podUID="49075769-3848-432c-8af5-7d1a4b07b13f" containerName="revoke-keys"
Oct 30 14:49:58.131063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.130993    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49075769-3848-432c-8af5-7d1a4b07b13f" containerName="revoke-keys"
Oct 30 14:49:58.131063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131013    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy"
Oct 30 14:49:58.131063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131020    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c09cc6a-b1c6-4dba-a559-1f957e3d3e4d" containerName="istio-proxy"
Oct 30 14:49:58.131063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131045    4530 cpu_manager.go:414] "containerMap: removing container" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy"
Oct 30 14:49:58.131063 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131052    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy"
Oct 30 14:49:58.131444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131077    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerName="istio-proxy"
Oct 30 14:49:58.131444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131085    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerName="istio-proxy"
Oct 30 14:49:58.131444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131203    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerName="istio-proxy"
Oct 30 14:49:58.131444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131220    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy"
Oct 30 14:49:58.131444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131231    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="49075769-3848-432c-8af5-7d1a4b07b13f" containerName="revoke-keys"
Oct 30 14:49:58.131444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131244    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ddd71b-1fbf-4359-b8a3-da951b41869b" containerName="istio-proxy"
Oct 30 14:49:58.131444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131256    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a06488b-fc85-45f1-bbf9-a40a39125e5c" containerName="istio-proxy"
Oct 30 14:49:58.131444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131267    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf51e1b-998e-4a06-aecd-80a7d28f7ab3" containerName="istio-proxy"
Oct 30 14:49:58.131444 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.131278    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="daef6f1c-c355-4357-aa4c-f94ba2619dd9" containerName="istio-proxy"
Oct 30 14:49:58.136378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.136353    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:49:58.153366 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.153334    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"]
Oct 30 14:49:58.164666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.164643    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:49:58.164845 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.164824    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:49:58.233178 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.233140    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-532c6e25-x26fm\" (UID: \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:49:58.234167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.233207    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-532c6e25-x26fm\" (UID: \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:49:58.234532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.234500    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jllfv\" (UniqueName: \"kubernetes.io/projected/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-kube-api-access-jllfv\") pod \"maas-api-revoke-sub-532c6e25-x26fm\" (UID: \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:49:58.337176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.337131    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jllfv\" (UniqueName: \"kubernetes.io/projected/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-kube-api-access-jllfv\") pod \"maas-api-revoke-sub-532c6e25-x26fm\" (UID: \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:49:58.337379 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.337243    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-532c6e25-x26fm\" (UID: \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:49:58.337379 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.337263    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-532c6e25-x26fm\" (UID: \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:49:58.337961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.337936    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-532c6e25-x26fm\" (UID: \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:49:58.340041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.340017    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-532c6e25-x26fm\" (UID: \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:49:58.364128 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.364097    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jllfv\" (UniqueName: \"kubernetes.io/projected/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-kube-api-access-jllfv\") pod \"maas-api-revoke-sub-532c6e25-x26fm\" (UID: \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:49:58.451965 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.451899    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:49:58.753121 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:49:58.753077    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c8a429_c99d_4a91_b4e5_e3f18274e20e.slice/crio-fa5f1ca89ff76d73c59c72b0fcb21d52120b1bdfce2acd61cd840e1069b8c4be WatchSource:0}: Error finding container fa5f1ca89ff76d73c59c72b0fcb21d52120b1bdfce2acd61cd840e1069b8c4be: Status 404 returned error can't find the container with id fa5f1ca89ff76d73c59c72b0fcb21d52120b1bdfce2acd61cd840e1069b8c4be
Oct 30 14:49:58.754189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.754156    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"]
Oct 30 14:49:58.829968 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:58.829910    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm" event={"ID":"e9c8a429-c99d-4a91-b4e5-e3f18274e20e","Type":"ContainerStarted","Data":"fa5f1ca89ff76d73c59c72b0fcb21d52120b1bdfce2acd61cd840e1069b8c4be"}
Oct 30 14:49:59.837605 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:59.837568    4530 generic.go:356] "Generic (PLEG): container finished" podID="e9c8a429-c99d-4a91-b4e5-e3f18274e20e" containerID="d02ad38ffff6bfa49d31abf1cb586c4a435e97acf060a51096892ee600e216d0" exitCode=0
Oct 30 14:49:59.838034 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:49:59.837654    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm" event={"ID":"e9c8a429-c99d-4a91-b4e5-e3f18274e20e","Type":"ContainerDied","Data":"d02ad38ffff6bfa49d31abf1cb586c4a435e97acf060a51096892ee600e216d0"}
Oct 30 14:50:00.468015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.467983    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"]
Oct 30 14:50:00.468268 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.468223    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76" podUID="fe54ee77-b40f-446a-9f20-55312643112c" containerName="maas-api" containerID="cri-o://2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b" gracePeriod=30
Oct 30 14:50:00.567323 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.567278    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main" containerID="cri-o://2822643bb78b9d408ad6c4c5246aed382fe8c9ae6e29bd56a08cb04323f1fd8a" gracePeriod=45
Oct 30 14:50:00.599565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.599482    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" containerID="cri-o://a754a11ca9215ab7589a6028f94a83926204b6759cbe9147bc00f3752c9e2870" gracePeriod=45
Oct 30 14:50:00.737603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.733779    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:50:00.846004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.845962    4530 generic.go:356] "Generic (PLEG): container finished" podID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerID="2822643bb78b9d408ad6c4c5246aed382fe8c9ae6e29bd56a08cb04323f1fd8a" exitCode=0
Oct 30 14:50:00.846489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.846037    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" event={"ID":"2893d528-7fcc-456f-bf7f-61f7d7f89b2a","Type":"ContainerDied","Data":"2822643bb78b9d408ad6c4c5246aed382fe8c9ae6e29bd56a08cb04323f1fd8a"}
Oct 30 14:50:00.848087 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.848059    4530 generic.go:356] "Generic (PLEG): container finished" podID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerID="a754a11ca9215ab7589a6028f94a83926204b6759cbe9147bc00f3752c9e2870" exitCode=0
Oct 30 14:50:00.848233 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.848129    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" event={"ID":"96aa8bcc-9208-4741-8cd4-f961070a462f","Type":"ContainerDied","Data":"a754a11ca9215ab7589a6028f94a83926204b6759cbe9147bc00f3752c9e2870"}
Oct 30 14:50:00.850286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.850263    4530 generic.go:356] "Generic (PLEG): container finished" podID="fe54ee77-b40f-446a-9f20-55312643112c" containerID="2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b" exitCode=0
Oct 30 14:50:00.850405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.850326    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"
Oct 30 14:50:00.850580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.850519    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76" event={"ID":"fe54ee77-b40f-446a-9f20-55312643112c","Type":"ContainerDied","Data":"2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b"}
Oct 30 14:50:00.850761 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.850741    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76" event={"ID":"fe54ee77-b40f-446a-9f20-55312643112c","Type":"ContainerDied","Data":"dc05b5de29502cd777e66e4ba5ad4447b784368c8ee4cfe71e07ec373a74f24c"}
Oct 30 14:50:00.851002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.850913    4530 scope.go:122] "RemoveContainer" containerID="2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b"
Oct 30 14:50:00.863287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.863265    4530 scope.go:122] "RemoveContainer" containerID="d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f"
Oct 30 14:50:00.864267 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.864240    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-metrics-certs\") pod \"fe54ee77-b40f-446a-9f20-55312643112c\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") "
Oct 30 14:50:00.864373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.864287    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-maas-api-tls\") pod \"fe54ee77-b40f-446a-9f20-55312643112c\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") "
Oct 30 14:50:00.864373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.864328    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe54ee77-b40f-446a-9f20-55312643112c-kube-api-access-p9h6m\" (UniqueName: \"kubernetes.io/projected/fe54ee77-b40f-446a-9f20-55312643112c-kube-api-access-p9h6m\") pod \"fe54ee77-b40f-446a-9f20-55312643112c\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") "
Oct 30 14:50:00.864513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.864391    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe54ee77-b40f-446a-9f20-55312643112c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe54ee77-b40f-446a-9f20-55312643112c-openshift-service-ca\") pod \"fe54ee77-b40f-446a-9f20-55312643112c\" (UID: \"fe54ee77-b40f-446a-9f20-55312643112c\") "
Oct 30 14:50:00.865311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.865192    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe54ee77-b40f-446a-9f20-55312643112c-openshift-service-ca" pod "fe54ee77-b40f-446a-9f20-55312643112c" (UID: "fe54ee77-b40f-446a-9f20-55312643112c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:00.867382 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.867352    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-maas-api-tls" pod "fe54ee77-b40f-446a-9f20-55312643112c" (UID: "fe54ee77-b40f-446a-9f20-55312643112c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:50:00.867490 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.867463    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-metrics-certs" pod "fe54ee77-b40f-446a-9f20-55312643112c" (UID: "fe54ee77-b40f-446a-9f20-55312643112c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:50:00.867820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.867794    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe54ee77-b40f-446a-9f20-55312643112c-kube-api-access-p9h6m" pod "fe54ee77-b40f-446a-9f20-55312643112c" (UID: "fe54ee77-b40f-446a-9f20-55312643112c"). InnerVolumeSpecName "kube-api-access-p9h6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:00.869750 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.869727    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:50:00.875179 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.875156    4530 scope.go:122] "RemoveContainer" containerID="2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b"
Oct 30 14:50:00.875434 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:50:00.875415    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b\": container with ID starting with 2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b not found: ID does not exist" containerID="2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b"
Oct 30 14:50:00.875499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.875442    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b"} err="failed to get container status \"2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b\": rpc error: code = NotFound desc = could not find container \"2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b\": container with ID starting with 2d422762ff05844c8fc257920f26290c903da0d0f97bb9782cf1739cad62b95b not found: ID does not exist"
Oct 30 14:50:00.875499 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.875458    4530 scope.go:122] "RemoveContainer" containerID="d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f"
Oct 30 14:50:00.875763 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:50:00.875744    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f\": container with ID starting with d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f not found: ID does not exist" containerID="d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f"
Oct 30 14:50:00.875822 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.875767    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f"} err="failed to get container status \"d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f\": rpc error: code = NotFound desc = could not find container \"d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f\": container with ID starting with d492bca9fde3c698fa99e076fb33efaeef0f53942dd8a6623c60370e2970473f not found: ID does not exist"
Oct 30 14:50:00.905076 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.900398    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:50:00.917141 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.917096    4530 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:50:00.917308 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.917197    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:50:00.966070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966042    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96aa8bcc-9208-4741-8cd4-f961070a462f-tls-certs\" (UniqueName: \"kubernetes.io/secret/96aa8bcc-9208-4741-8cd4-f961070a462f-tls-certs\") pod \"96aa8bcc-9208-4741-8cd4-f961070a462f\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") "
Oct 30 14:50:00.966237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966125    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-model-cache\") pod \"96aa8bcc-9208-4741-8cd4-f961070a462f\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") "
Oct 30 14:50:00.966237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966187    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96aa8bcc-9208-4741-8cd4-f961070a462f-kube-api-access-rw7pl\" (UniqueName: \"kubernetes.io/projected/96aa8bcc-9208-4741-8cd4-f961070a462f-kube-api-access-rw7pl\") pod \"96aa8bcc-9208-4741-8cd4-f961070a462f\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") "
Oct 30 14:50:00.966237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966227    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-dshm\") pod \"96aa8bcc-9208-4741-8cd4-f961070a462f\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") "
Oct 30 14:50:00.966418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966269    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-home\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-home\") pod \"96aa8bcc-9208-4741-8cd4-f961070a462f\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") "
Oct 30 14:50:00.966418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966298    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-tmp-dir\") pod \"96aa8bcc-9208-4741-8cd4-f961070a462f\" (UID: \"96aa8bcc-9208-4741-8cd4-f961070a462f\") "
Oct 30 14:50:00.966527 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966422    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-model-cache" pod "96aa8bcc-9208-4741-8cd4-f961070a462f" (UID: "96aa8bcc-9208-4741-8cd4-f961070a462f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:00.966652 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966632    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-tmp-dir" pod "96aa8bcc-9208-4741-8cd4-f961070a462f" (UID: "96aa8bcc-9208-4741-8cd4-f961070a462f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:00.966777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966759    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:00.966836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966781    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe54ee77-b40f-446a-9f20-55312643112c-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:00.966836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966796    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9h6m\" (UniqueName: \"kubernetes.io/projected/fe54ee77-b40f-446a-9f20-55312643112c-kube-api-access-p9h6m\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:00.966836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966813    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe54ee77-b40f-446a-9f20-55312643112c-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:00.966836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966814    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-home" pod "96aa8bcc-9208-4741-8cd4-f961070a462f" (UID: "96aa8bcc-9208-4741-8cd4-f961070a462f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:00.966836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966827    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:00.967020 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.966841    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:00.968839 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.968807    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96aa8bcc-9208-4741-8cd4-f961070a462f-kube-api-access-rw7pl" pod "96aa8bcc-9208-4741-8cd4-f961070a462f" (UID: "96aa8bcc-9208-4741-8cd4-f961070a462f"). InnerVolumeSpecName "kube-api-access-rw7pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:00.968952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.968901    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96aa8bcc-9208-4741-8cd4-f961070a462f-tls-certs" pod "96aa8bcc-9208-4741-8cd4-f961070a462f" (UID: "96aa8bcc-9208-4741-8cd4-f961070a462f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:50:00.969041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:00.969023    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-dshm" pod "96aa8bcc-9208-4741-8cd4-f961070a462f" (UID: "96aa8bcc-9208-4741-8cd4-f961070a462f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:01.031793 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.031765    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:50:01.067307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.067277    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-home\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-home\") pod \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") "
Oct 30 14:50:01.067493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.067322    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-model-cache\") pod \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") "
Oct 30 14:50:01.067493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.067410    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tls-certs\") pod \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") "
Oct 30 14:50:01.067493 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.067443    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-kube-api-access-vttht\" (UniqueName: \"kubernetes.io/projected/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-kube-api-access-vttht\") pod \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") "
Oct 30 14:50:01.067704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.067613    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tmp-dir\") pod \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") "
Oct 30 14:50:01.067704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.067587    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-home" pod "2893d528-7fcc-456f-bf7f-61f7d7f89b2a" (UID: "2893d528-7fcc-456f-bf7f-61f7d7f89b2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:01.067704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.067640    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-model-cache" pod "2893d528-7fcc-456f-bf7f-61f7d7f89b2a" (UID: "2893d528-7fcc-456f-bf7f-61f7d7f89b2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:01.067883 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.067671    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-dshm\") pod \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\" (UID: \"2893d528-7fcc-456f-bf7f-61f7d7f89b2a\") "
Oct 30 14:50:01.067883 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.067779    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tmp-dir" pod "2893d528-7fcc-456f-bf7f-61f7d7f89b2a" (UID: "2893d528-7fcc-456f-bf7f-61f7d7f89b2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:01.068094 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.068062    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw7pl\" (UniqueName: \"kubernetes.io/projected/96aa8bcc-9208-4741-8cd4-f961070a462f-kube-api-access-rw7pl\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.068094 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.068087    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.068269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.068099    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96aa8bcc-9208-4741-8cd4-f961070a462f-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.068269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.068111    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.068269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.068123    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96aa8bcc-9208-4741-8cd4-f961070a462f-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.068269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.068133    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.068269 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.068144    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.069614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.069591    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tls-certs" pod "2893d528-7fcc-456f-bf7f-61f7d7f89b2a" (UID: "2893d528-7fcc-456f-bf7f-61f7d7f89b2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:50:01.069726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.069624    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-kube-api-access-vttht" pod "2893d528-7fcc-456f-bf7f-61f7d7f89b2a" (UID: "2893d528-7fcc-456f-bf7f-61f7d7f89b2a"). InnerVolumeSpecName "kube-api-access-vttht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:01.069726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.069624    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-dshm" pod "2893d528-7fcc-456f-bf7f-61f7d7f89b2a" (UID: "2893d528-7fcc-456f-bf7f-61f7d7f89b2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:01.168832 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.168802    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-openshift-ca\") pod \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\" (UID: \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\") "
Oct 30 14:50:01.168986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.168859    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-kube-api-access-jllfv\" (UniqueName: \"kubernetes.io/projected/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-kube-api-access-jllfv\") pod \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\" (UID: \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\") "
Oct 30 14:50:01.168986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.168895    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-xks-ca\") pod \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\" (UID: \"e9c8a429-c99d-4a91-b4e5-e3f18274e20e\") "
Oct 30 14:50:01.169172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.169148    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-openshift-ca" pod "e9c8a429-c99d-4a91-b4e5-e3f18274e20e" (UID: "e9c8a429-c99d-4a91-b4e5-e3f18274e20e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:01.169231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.169212    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.169278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.169239    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vttht\" (UniqueName: \"kubernetes.io/projected/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-kube-api-access-vttht\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.169278 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.169254    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2893d528-7fcc-456f-bf7f-61f7d7f89b2a-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.170900 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.170869    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-kube-api-access-jllfv" pod "e9c8a429-c99d-4a91-b4e5-e3f18274e20e" (UID: "e9c8a429-c99d-4a91-b4e5-e3f18274e20e"). InnerVolumeSpecName "kube-api-access-jllfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:01.171031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.170911    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-xks-ca" pod "e9c8a429-c99d-4a91-b4e5-e3f18274e20e" (UID: "e9c8a429-c99d-4a91-b4e5-e3f18274e20e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:50:01.196897 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.196865    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"]
Oct 30 14:50:01.203918 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.203896    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-228454-d975bf7cd-q2c76"]
Oct 30 14:50:01.270297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.270268    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.270297 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.270298    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.270517 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.270312    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jllfv\" (UniqueName: \"kubernetes.io/projected/e9c8a429-c99d-4a91-b4e5-e3f18274e20e-kube-api-access-jllfv\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:01.787724 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.787623    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:50:01.857555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.857512    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"
Oct 30 14:50:01.857555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.857516    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp" event={"ID":"96aa8bcc-9208-4741-8cd4-f961070a462f","Type":"ContainerDied","Data":"e10fd1f1bdfb96801eee158908730faf964dd7d40b311336aaa280cd698a7993"}
Oct 30 14:50:01.858075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.857587    4530 scope.go:122] "RemoveContainer" containerID="a754a11ca9215ab7589a6028f94a83926204b6759cbe9147bc00f3752c9e2870"
Oct 30 14:50:01.860644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.860624    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"
Oct 30 14:50:01.860745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.860658    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj" event={"ID":"2893d528-7fcc-456f-bf7f-61f7d7f89b2a","Type":"ContainerDied","Data":"f501b8518befc2f1f3449a0b72ecaa6c2e15b2bb30a6809dd1eef92d3d24cc6e"}
Oct 30 14:50:01.862194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.862173    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm" event={"ID":"e9c8a429-c99d-4a91-b4e5-e3f18274e20e","Type":"ContainerDied","Data":"fa5f1ca89ff76d73c59c72b0fcb21d52120b1bdfce2acd61cd840e1069b8c4be"}
Oct 30 14:50:01.862312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.862211    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa5f1ca89ff76d73c59c72b0fcb21d52120b1bdfce2acd61cd840e1069b8c4be"
Oct 30 14:50:01.862312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.862259    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"
Oct 30 14:50:01.869128 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.869109    4530 scope.go:122] "RemoveContainer" containerID="2822643bb78b9d408ad6c4c5246aed382fe8c9ae6e29bd56a08cb04323f1fd8a"
Oct 30 14:50:01.873997 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.873975    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"]
Oct 30 14:50:01.886282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.886263    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0efe43-kz6ht"]
Oct 30 14:50:01.927492 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.927455    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"]
Oct 30 14:50:01.932390 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.932358    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/simulator-w0-2df0ab-kserve-84d768799d-btlbp"]
Oct 30 14:50:01.991448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.991418    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"]
Oct 30 14:50:01.998304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:01.998274    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/distinct-2-w0-2df0ab-kserve-7d5c5bd5-bmxsj"]
Oct 30 14:50:02.118674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.118649    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:50:02.280872 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.280838    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd9b019-951d-47cf-a812-946f4e10c8ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd9b019-951d-47cf-a812-946f4e10c8ba-istiod-ca-cert\") pod \"7bd9b019-951d-47cf-a812-946f4e10c8ba\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") "
Oct 30 14:50:02.281073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.280916    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-credential-socket\") pod \"7bd9b019-951d-47cf-a812-946f4e10c8ba\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") "
Oct 30 14:50:02.281073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281004    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-certs\") pod \"7bd9b019-951d-47cf-a812-946f4e10c8ba\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") "
Oct 30 14:50:02.281073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281030    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-podinfo\") pod \"7bd9b019-951d-47cf-a812-946f4e10c8ba\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") "
Oct 30 14:50:02.281073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281065    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-data\") pod \"7bd9b019-951d-47cf-a812-946f4e10c8ba\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") "
Oct 30 14:50:02.281300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281084    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-socket\") pod \"7bd9b019-951d-47cf-a812-946f4e10c8ba\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") "
Oct 30 14:50:02.281300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281125    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-envoy\") pod \"7bd9b019-951d-47cf-a812-946f4e10c8ba\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") "
Oct 30 14:50:02.281300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281165    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-kube-api-access-bc7q9\" (UniqueName: \"kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-kube-api-access-bc7q9\") pod \"7bd9b019-951d-47cf-a812-946f4e10c8ba\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") "
Oct 30 14:50:02.281300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281211    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-token\" (UniqueName: \"kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-token\") pod \"7bd9b019-951d-47cf-a812-946f4e10c8ba\" (UID: \"7bd9b019-951d-47cf-a812-946f4e10c8ba\") "
Oct 30 14:50:02.281300 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281229    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-credential-socket" pod "7bd9b019-951d-47cf-a812-946f4e10c8ba" (UID: "7bd9b019-951d-47cf-a812-946f4e10c8ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:02.281577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281378    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-data" pod "7bd9b019-951d-47cf-a812-946f4e10c8ba" (UID: "7bd9b019-951d-47cf-a812-946f4e10c8ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:02.281577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281393    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-certs" pod "7bd9b019-951d-47cf-a812-946f4e10c8ba" (UID: "7bd9b019-951d-47cf-a812-946f4e10c8ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:02.281577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281402    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd9b019-951d-47cf-a812-946f4e10c8ba-istiod-ca-cert" pod "7bd9b019-951d-47cf-a812-946f4e10c8ba" (UID: "7bd9b019-951d-47cf-a812-946f4e10c8ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:02.281749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281599    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:02.281749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281622    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:02.281749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281639    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:02.281749 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281652    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd9b019-951d-47cf-a812-946f4e10c8ba-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:02.282113 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.281753    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-socket" pod "7bd9b019-951d-47cf-a812-946f4e10c8ba" (UID: "7bd9b019-951d-47cf-a812-946f4e10c8ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:02.284073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.284046    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-envoy" pod "7bd9b019-951d-47cf-a812-946f4e10c8ba" (UID: "7bd9b019-951d-47cf-a812-946f4e10c8ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:02.284176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.284072    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-kube-api-access-bc7q9" pod "7bd9b019-951d-47cf-a812-946f4e10c8ba" (UID: "7bd9b019-951d-47cf-a812-946f4e10c8ba"). InnerVolumeSpecName "kube-api-access-bc7q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:02.284176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.284080    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-podinfo" pod "7bd9b019-951d-47cf-a812-946f4e10c8ba" (UID: "7bd9b019-951d-47cf-a812-946f4e10c8ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:50:02.284176 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.284101    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-token" pod "7bd9b019-951d-47cf-a812-946f4e10c8ba" (UID: "7bd9b019-951d-47cf-a812-946f4e10c8ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:02.383084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.383001    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:02.383084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.383033    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:02.383084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.383043    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:02.383084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.383052    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc7q9\" (UniqueName: \"kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-kube-api-access-bc7q9\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:02.383084 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.383060    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd9b019-951d-47cf-a812-946f4e10c8ba-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:02.871450 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.871415    4530 generic.go:356] "Generic (PLEG): container finished" podID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerID="061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b" exitCode=0
Oct 30 14:50:02.871931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.871461    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"
Oct 30 14:50:02.871931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.871480    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" event={"ID":"7bd9b019-951d-47cf-a812-946f4e10c8ba","Type":"ContainerDied","Data":"061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b"}
Oct 30 14:50:02.871931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.871522    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h" event={"ID":"7bd9b019-951d-47cf-a812-946f4e10c8ba","Type":"ContainerDied","Data":"3d99e83e0b751cac9bba4c41abe76d5e06bda8bbfd35c0cc46fbe0f99faad02c"}
Oct 30 14:50:02.871931 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.871564    4530 scope.go:122] "RemoveContainer" containerID="061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b"
Oct 30 14:50:02.882172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.882152    4530 scope.go:122] "RemoveContainer" containerID="eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4"
Oct 30 14:50:02.891638 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.891616    4530 scope.go:122] "RemoveContainer" containerID="061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b"
Oct 30 14:50:02.891923 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:50:02.891891    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b\": container with ID starting with 061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b not found: ID does not exist" containerID="061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b"
Oct 30 14:50:02.891986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.891932    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b"} err="failed to get container status \"061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b\": rpc error: code = NotFound desc = could not find container \"061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b\": container with ID starting with 061dc7c970830474c826acfb2a213e21b021c215916c4657bfc1a7b36617e63b not found: ID does not exist"
Oct 30 14:50:02.891986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.891948    4530 scope.go:122] "RemoveContainer" containerID="eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4"
Oct 30 14:50:02.892197 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:50:02.892178    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4\": container with ID starting with eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4 not found: ID does not exist" containerID="eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4"
Oct 30 14:50:02.892236 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.892203    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4"} err="failed to get container status \"eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4\": rpc error: code = NotFound desc = could not find container \"eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4\": container with ID starting with eb574bccd3c22f8efa3ba13a6b158f715486fbddccaeaf11b4ac0efd0b44abd4 not found: ID does not exist"
Oct 30 14:50:02.934378 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.934342    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"]
Oct 30 14:50:02.948618 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:02.948583    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-bzj5h"]
Oct 30 14:50:03.113826 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:03.113789    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" path="/var/lib/kubelet/pods/2893d528-7fcc-456f-bf7f-61f7d7f89b2a/volumes"
Oct 30 14:50:03.114207 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:03.114195    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49075769-3848-432c-8af5-7d1a4b07b13f" path="/var/lib/kubelet/pods/49075769-3848-432c-8af5-7d1a4b07b13f/volumes"
Oct 30 14:50:03.114642 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:03.114627    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" path="/var/lib/kubelet/pods/7bd9b019-951d-47cf-a812-946f4e10c8ba/volumes"
Oct 30 14:50:03.115357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:03.115342    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" path="/var/lib/kubelet/pods/96aa8bcc-9208-4741-8cd4-f961070a462f/volumes"
Oct 30 14:50:03.115737 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:03.115725    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe54ee77-b40f-446a-9f20-55312643112c" path="/var/lib/kubelet/pods/fe54ee77-b40f-446a-9f20-55312643112c/volumes"
Oct 30 14:50:03.203629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:03.196744    4530 scope.go:122] "RemoveContainer" containerID="35ad71c79a7ea1ac1085520dba1645871f8da5a06524074c3aed2e76b38a690a"
Oct 30 14:50:04.984732 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:04.984695    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:50:05.448758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:05.448714    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj" podUID="f4ba7a80-b30b-40b3-a209-674a54c6f096" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:50:08.018867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:08.018826    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"]
Oct 30 14:50:09.187080 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187044    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"]
Oct 30 14:50:09.187491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187451    4530 cpu_manager.go:414] "containerMap: removing container" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main"
Oct 30 14:50:09.187491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187462    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main"
Oct 30 14:50:09.187491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187473    4530 cpu_manager.go:414] "containerMap: removing container" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy"
Oct 30 14:50:09.187491 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187478    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy"
Oct 30 14:50:09.187645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187494    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e9c8a429-c99d-4a91-b4e5-e3f18274e20e" containerName="revoke-keys"
Oct 30 14:50:09.187645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187500    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c8a429-c99d-4a91-b4e5-e3f18274e20e" containerName="revoke-keys"
Oct 30 14:50:09.187645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187512    4530 cpu_manager.go:414] "containerMap: removing container" podUID="fe54ee77-b40f-446a-9f20-55312643112c" containerName="maas-api"
Oct 30 14:50:09.187645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187517    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe54ee77-b40f-446a-9f20-55312643112c" containerName="maas-api"
Oct 30 14:50:09.187645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187525    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy"
Oct 30 14:50:09.187645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187529    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy"
Oct 30 14:50:09.187645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187563    4530 cpu_manager.go:414] "containerMap: removing container" podUID="fe54ee77-b40f-446a-9f20-55312643112c" containerName="maas-api"
Oct 30 14:50:09.187645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187568    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe54ee77-b40f-446a-9f20-55312643112c" containerName="maas-api"
Oct 30 14:50:09.187645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187577    4530 cpu_manager.go:414] "containerMap: removing container" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main"
Oct 30 14:50:09.187645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187582    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main"
Oct 30 14:50:09.187645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187593    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerName="istio-proxy"
Oct 30 14:50:09.187645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187598    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerName="istio-proxy"
Oct 30 14:50:09.188010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187660    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac20408-a58d-4bc9-8526-b86fe83d9020" containerName="istio-proxy"
Oct 30 14:50:09.188010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187670    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="2893d528-7fcc-456f-bf7f-61f7d7f89b2a" containerName="main"
Oct 30 14:50:09.188010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187680    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7bf62ef-287a-4ede-95e9-a0c8d8354141" containerName="istio-proxy"
Oct 30 14:50:09.188010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187688    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c8a429-c99d-4a91-b4e5-e3f18274e20e" containerName="revoke-keys"
Oct 30 14:50:09.188010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187695    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe54ee77-b40f-446a-9f20-55312643112c" containerName="maas-api"
Oct 30 14:50:09.188010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187700    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy"
Oct 30 14:50:09.188010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187706    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe54ee77-b40f-446a-9f20-55312643112c" containerName="maas-api"
Oct 30 14:50:09.188010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187712    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy"
Oct 30 14:50:09.188010 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.187717    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="96aa8bcc-9208-4741-8cd4-f961070a462f" containerName="main"
Oct 30 14:50:09.193021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.192997    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:09.216325 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.216295    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"]
Oct 30 14:50:09.250478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.250438    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-342ce888-mwhp8\" (UID: \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:09.250659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.250497    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-342ce888-mwhp8\" (UID: \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:09.250659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.250594    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m64fq\" (UniqueName: \"kubernetes.io/projected/94d586e5-be6a-4560-8e91-3fcf9fd13e04-kube-api-access-m64fq\") pod \"maas-api-revoke-sub-342ce888-mwhp8\" (UID: \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:09.285259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.285222    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:50:09.285259 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.285231    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:50:09.351317 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.351282    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-342ce888-mwhp8\" (UID: \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:09.351517 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.351356    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-342ce888-mwhp8\" (UID: \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:09.351517 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.351374    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m64fq\" (UniqueName: \"kubernetes.io/projected/94d586e5-be6a-4560-8e91-3fcf9fd13e04-kube-api-access-m64fq\") pod \"maas-api-revoke-sub-342ce888-mwhp8\" (UID: \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:09.352108 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.352076    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-342ce888-mwhp8\" (UID: \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:09.353796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.353775    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-342ce888-mwhp8\" (UID: \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:09.378809 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.378784    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m64fq\" (UniqueName: \"kubernetes.io/projected/94d586e5-be6a-4560-8e91-3fcf9fd13e04-kube-api-access-m64fq\") pod \"maas-api-revoke-sub-342ce888-mwhp8\" (UID: \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:09.507008 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.506915    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:09.810736 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.810705    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"]
Oct 30 14:50:09.811008 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:50:09.810977    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d586e5_be6a_4560_8e91_3fcf9fd13e04.slice/crio-1c1a37687dab9a3ea5bd05be0ecb21fd119aae3edeb32b3dd067a82813494fa2 WatchSource:0}: Error finding container 1c1a37687dab9a3ea5bd05be0ecb21fd119aae3edeb32b3dd067a82813494fa2: Status 404 returned error can't find the container with id 1c1a37687dab9a3ea5bd05be0ecb21fd119aae3edeb32b3dd067a82813494fa2
Oct 30 14:50:09.912286 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:09.912250    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8" event={"ID":"94d586e5-be6a-4560-8e91-3fcf9fd13e04","Type":"ContainerStarted","Data":"1c1a37687dab9a3ea5bd05be0ecb21fd119aae3edeb32b3dd067a82813494fa2"}
Oct 30 14:50:10.550229 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:10.550195    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"]
Oct 30 14:50:10.550645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:10.550406    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" podUID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" containerName="istio-proxy" containerID="cri-o://ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112" gracePeriod=30
Oct 30 14:50:10.919255 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:10.919174    4530 generic.go:356] "Generic (PLEG): container finished" podID="94d586e5-be6a-4560-8e91-3fcf9fd13e04" containerID="16dc91ffeb88b0c2684ac115facb30fbe8bd2d60c8238fa744a5c9265cfe1e77" exitCode=0
Oct 30 14:50:10.919396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:10.919236    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8" event={"ID":"94d586e5-be6a-4560-8e91-3fcf9fd13e04","Type":"ContainerDied","Data":"16dc91ffeb88b0c2684ac115facb30fbe8bd2d60c8238fa744a5c9265cfe1e77"}
Oct 30 14:50:11.228748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.228662    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"]
Oct 30 14:50:11.229113 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.229099    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy"
Oct 30 14:50:11.229172 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.229116    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd9b019-951d-47cf-a812-946f4e10c8ba" containerName="istio-proxy"
Oct 30 14:50:11.232455 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.232432    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:11.251917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.251889    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"]
Oct 30 14:50:11.269671 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.269641    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp\" (UID: \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:11.269836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.269698    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp\" (UID: \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:11.269836 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.269784    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4575\" (UniqueName: \"kubernetes.io/projected/7a8981e5-9b43-41b9-9206-f47a5929d4ff-kube-api-access-r4575\") pod \"maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp\" (UID: \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:11.371403 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:50:11.371357    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:50:11.371604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.371453    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4575\" (UniqueName: \"kubernetes.io/projected/7a8981e5-9b43-41b9-9206-f47a5929d4ff-kube-api-access-r4575\") pod \"maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp\" (UID: \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:11.371604 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.371553    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp\" (UID: \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:11.371707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.371622    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp\" (UID: \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:11.372295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.372266    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp\" (UID: \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:11.373954 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.373935    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp\" (UID: \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:11.398502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.398474    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4575\" (UniqueName: \"kubernetes.io/projected/7a8981e5-9b43-41b9-9206-f47a5929d4ff-kube-api-access-r4575\") pod \"maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp\" (UID: \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:11.545860 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.545766    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:11.844842 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:50:11.844812    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8981e5_9b43_41b9_9206_f47a5929d4ff.slice/crio-3507b9b9d52436ad76e5575b9181259c43cc35cfbe713678f7ee1d999c7211b8 WatchSource:0}: Error finding container 3507b9b9d52436ad76e5575b9181259c43cc35cfbe713678f7ee1d999c7211b8: Status 404 returned error can't find the container with id 3507b9b9d52436ad76e5575b9181259c43cc35cfbe713678f7ee1d999c7211b8
Oct 30 14:50:11.846156 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.846127    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"]
Oct 30 14:50:11.926023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:11.925983    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp" event={"ID":"7a8981e5-9b43-41b9-9206-f47a5929d4ff","Type":"ContainerStarted","Data":"3507b9b9d52436ad76e5575b9181259c43cc35cfbe713678f7ee1d999c7211b8"}
Oct 30 14:50:12.060132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.060103    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:12.077347 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.077318    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d586e5-be6a-4560-8e91-3fcf9fd13e04-kube-api-access-m64fq\" (UniqueName: \"kubernetes.io/projected/94d586e5-be6a-4560-8e91-3fcf9fd13e04-kube-api-access-m64fq\") pod \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\" (UID: \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\") "
Oct 30 14:50:12.077502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.077435    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-xks-ca\") pod \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\" (UID: \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\") "
Oct 30 14:50:12.077502 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.077473    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-openshift-ca\") pod \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\" (UID: \"94d586e5-be6a-4560-8e91-3fcf9fd13e04\") "
Oct 30 14:50:12.077884 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.077856    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-openshift-ca" pod "94d586e5-be6a-4560-8e91-3fcf9fd13e04" (UID: "94d586e5-be6a-4560-8e91-3fcf9fd13e04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:12.079473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.079453    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-xks-ca" pod "94d586e5-be6a-4560-8e91-3fcf9fd13e04" (UID: "94d586e5-be6a-4560-8e91-3fcf9fd13e04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:50:12.079570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.079489    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d586e5-be6a-4560-8e91-3fcf9fd13e04-kube-api-access-m64fq" pod "94d586e5-be6a-4560-8e91-3fcf9fd13e04" (UID: "94d586e5-be6a-4560-8e91-3fcf9fd13e04"). InnerVolumeSpecName "kube-api-access-m64fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:12.179195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.179118    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:12.179195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.179145    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d586e5-be6a-4560-8e91-3fcf9fd13e04-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:12.179195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.179156    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m64fq\" (UniqueName: \"kubernetes.io/projected/94d586e5-be6a-4560-8e91-3fcf9fd13e04-kube-api-access-m64fq\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:12.933260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.933225    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"
Oct 30 14:50:12.933260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.933241    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8" event={"ID":"94d586e5-be6a-4560-8e91-3fcf9fd13e04","Type":"ContainerDied","Data":"1c1a37687dab9a3ea5bd05be0ecb21fd119aae3edeb32b3dd067a82813494fa2"}
Oct 30 14:50:12.933797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.933283    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c1a37687dab9a3ea5bd05be0ecb21fd119aae3edeb32b3dd067a82813494fa2"
Oct 30 14:50:12.934935 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.934906    4530 generic.go:356] "Generic (PLEG): container finished" podID="7a8981e5-9b43-41b9-9206-f47a5929d4ff" containerID="69d56bf086c3c7e188bc414b9d3847c8174b35ebde1bbeb44a62b92ae670e918" exitCode=0
Oct 30 14:50:12.935104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:12.934994    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp" event={"ID":"7a8981e5-9b43-41b9-9206-f47a5929d4ff","Type":"ContainerDied","Data":"69d56bf086c3c7e188bc414b9d3847c8174b35ebde1bbeb44a62b92ae670e918"}
Oct 30 14:50:13.020057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:13.019976    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2" podUID="e54a8e55-53ec-4a30-a4ec-bc39634e5a63" containerName="payload-processing" containerID="cri-o://ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2" gracePeriod=25
Oct 30 14:50:14.084695 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.084669    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:14.095853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.095826    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8981e5-9b43-41b9-9206-f47a5929d4ff-kube-api-access-r4575\" (UniqueName: \"kubernetes.io/projected/7a8981e5-9b43-41b9-9206-f47a5929d4ff-kube-api-access-r4575\") pod \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\" (UID: \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\") "
Oct 30 14:50:14.096016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.095914    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-openshift-ca\") pod \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\" (UID: \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\") "
Oct 30 14:50:14.096016 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.096004    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-xks-ca\") pod \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\" (UID: \"7a8981e5-9b43-41b9-9206-f47a5929d4ff\") "
Oct 30 14:50:14.096245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.096220    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-openshift-ca" pod "7a8981e5-9b43-41b9-9206-f47a5929d4ff" (UID: "7a8981e5-9b43-41b9-9206-f47a5929d4ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:14.097952 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.097928    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8981e5-9b43-41b9-9206-f47a5929d4ff-kube-api-access-r4575" pod "7a8981e5-9b43-41b9-9206-f47a5929d4ff" (UID: "7a8981e5-9b43-41b9-9206-f47a5929d4ff"). InnerVolumeSpecName "kube-api-access-r4575". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:14.098056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.097971    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-xks-ca" pod "7a8981e5-9b43-41b9-9206-f47a5929d4ff" (UID: "7a8981e5-9b43-41b9-9206-f47a5929d4ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:50:14.197360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.197284    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:14.197360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.197322    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4575\" (UniqueName: \"kubernetes.io/projected/7a8981e5-9b43-41b9-9206-f47a5929d4ff-kube-api-access-r4575\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:14.197360 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.197338    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8981e5-9b43-41b9-9206-f47a5929d4ff-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:14.949876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.949843    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"
Oct 30 14:50:14.950073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.949885    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp" event={"ID":"7a8981e5-9b43-41b9-9206-f47a5929d4ff","Type":"ContainerDied","Data":"3507b9b9d52436ad76e5575b9181259c43cc35cfbe713678f7ee1d999c7211b8"}
Oct 30 14:50:14.950073 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:14.949947    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3507b9b9d52436ad76e5575b9181259c43cc35cfbe713678f7ee1d999c7211b8"
Oct 30 14:50:15.448553 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.448513    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj" podUID="f4ba7a80-b30b-40b3-a209-674a54c6f096" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:50:15.800451 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.800424    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:50:15.912185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912055    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-token\" (UniqueName: \"kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-token\") pod \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") "
Oct 30 14:50:15.912185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912121    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-credential-socket\") pod \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") "
Oct 30 14:50:15.912185 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912152    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-podinfo\") pod \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") "
Oct 30 14:50:15.912531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912299    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-envoy\") pod \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") "
Oct 30 14:50:15.912531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912351    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-data\") pod \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") "
Oct 30 14:50:15.912531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912391    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-socket\") pod \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") "
Oct 30 14:50:15.912531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912421    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istiod-ca-cert\") pod \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") "
Oct 30 14:50:15.912531 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912493    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-kube-api-access-m8qc9\" (UniqueName: \"kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-kube-api-access-m8qc9\") pod \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") "
Oct 30 14:50:15.912847 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912562    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-credential-socket" pod "f2c92412-886d-4c69-9c8e-3cf6fa1a344c" (UID: "f2c92412-886d-4c69-9c8e-3cf6fa1a344c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:15.912847 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912575    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-certs\") pod \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\" (UID: \"f2c92412-886d-4c69-9c8e-3cf6fa1a344c\") "
Oct 30 14:50:15.912847 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912683    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-data" pod "f2c92412-886d-4c69-9c8e-3cf6fa1a344c" (UID: "f2c92412-886d-4c69-9c8e-3cf6fa1a344c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:15.913015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912971    4530 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-credential-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:15.913015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.912994    4530 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-data\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:15.913116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.913032    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istiod-ca-cert" pod "f2c92412-886d-4c69-9c8e-3cf6fa1a344c" (UID: "f2c92412-886d-4c69-9c8e-3cf6fa1a344c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:15.913380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.913231    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-socket" pod "f2c92412-886d-4c69-9c8e-3cf6fa1a344c" (UID: "f2c92412-886d-4c69-9c8e-3cf6fa1a344c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:15.913380 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.913303    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-certs" pod "f2c92412-886d-4c69-9c8e-3cf6fa1a344c" (UID: "f2c92412-886d-4c69-9c8e-3cf6fa1a344c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:15.915280 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.915254    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-envoy" pod "f2c92412-886d-4c69-9c8e-3cf6fa1a344c" (UID: "f2c92412-886d-4c69-9c8e-3cf6fa1a344c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:50:15.915728 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.915690    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-kube-api-access-m8qc9" pod "f2c92412-886d-4c69-9c8e-3cf6fa1a344c" (UID: "f2c92412-886d-4c69-9c8e-3cf6fa1a344c"). InnerVolumeSpecName "kube-api-access-m8qc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:15.916421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.916394    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-token" pod "f2c92412-886d-4c69-9c8e-3cf6fa1a344c" (UID: "f2c92412-886d-4c69-9c8e-3cf6fa1a344c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:15.916514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.916479    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-podinfo" pod "f2c92412-886d-4c69-9c8e-3cf6fa1a344c" (UID: "f2c92412-886d-4c69-9c8e-3cf6fa1a344c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:50:15.957192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.957165    4530 generic.go:356] "Generic (PLEG): container finished" podID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" containerID="ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112" exitCode=0
Oct 30 14:50:15.957318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.957214    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"
Oct 30 14:50:15.957318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.957249    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" event={"ID":"f2c92412-886d-4c69-9c8e-3cf6fa1a344c","Type":"ContainerDied","Data":"ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112"}
Oct 30 14:50:15.957318 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.957303    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js" event={"ID":"f2c92412-886d-4c69-9c8e-3cf6fa1a344c","Type":"ContainerDied","Data":"a86f12315e045ae242cf0de160b3337a0fdfd77af36ea97534cb6a169f6c3cff"}
Oct 30 14:50:15.957473 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.957325    4530 scope.go:122] "RemoveContainer" containerID="ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112"
Oct 30 14:50:15.967468 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.967448    4530 scope.go:122] "RemoveContainer" containerID="ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112"
Oct 30 14:50:15.967754 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:50:15.967735    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112\": container with ID starting with ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112 not found: ID does not exist" containerID="ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112"
Oct 30 14:50:15.967815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:15.967761    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112"} err="failed to get container status \"ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112\": rpc error: code = NotFound desc = could not find container \"ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112\": container with ID starting with ed60b5086c91b6d8cf308c1833a75ee6c40bcb0eb7f0e7dc73169fef80d69112 not found: ID does not exist"
Oct 30 14:50:16.014057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.013977    4530 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-token\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:16.014057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.014007    4530 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-podinfo\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:16.014057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.014016    4530 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istio-envoy\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:16.014057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.014025    4530 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-socket\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:16.014057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.014034    4530 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-istiod-ca-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:16.014057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.014043    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8qc9\" (UniqueName: \"kubernetes.io/projected/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-kube-api-access-m8qc9\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:16.014057 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.014051    4530 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c92412-886d-4c69-9c8e-3cf6fa1a344c-workload-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:16.022525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.022498    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"]
Oct 30 14:50:16.031697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.031669    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-nz2js"]
Oct 30 14:50:16.493903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.493877    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"
Oct 30 14:50:16.519614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.518346    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ba7a80-b30b-40b3-a209-674a54c6f096-kube-api-access-wzjjr\" (UniqueName: \"kubernetes.io/projected/f4ba7a80-b30b-40b3-a209-674a54c6f096-kube-api-access-wzjjr\") pod \"f4ba7a80-b30b-40b3-a209-674a54c6f096\" (UID: \"f4ba7a80-b30b-40b3-a209-674a54c6f096\") "
Oct 30 14:50:16.519614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.518513    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ba7a80-b30b-40b3-a209-674a54c6f096-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ba7a80-b30b-40b3-a209-674a54c6f096-plugins-config-volume\") pod \"f4ba7a80-b30b-40b3-a209-674a54c6f096\" (UID: \"f4ba7a80-b30b-40b3-a209-674a54c6f096\") "
Oct 30 14:50:16.519614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.519491    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ba7a80-b30b-40b3-a209-674a54c6f096-plugins-config-volume" pod "f4ba7a80-b30b-40b3-a209-674a54c6f096" (UID: "f4ba7a80-b30b-40b3-a209-674a54c6f096"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:16.522072 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.522010    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ba7a80-b30b-40b3-a209-674a54c6f096-kube-api-access-wzjjr" pod "f4ba7a80-b30b-40b3-a209-674a54c6f096" (UID: "f4ba7a80-b30b-40b3-a209-674a54c6f096"). InnerVolumeSpecName "kube-api-access-wzjjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:16.619855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.619821    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ba7a80-b30b-40b3-a209-674a54c6f096-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:16.619855 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.619849    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzjjr\" (UniqueName: \"kubernetes.io/projected/f4ba7a80-b30b-40b3-a209-674a54c6f096-kube-api-access-wzjjr\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:16.966002 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.965957    4530 generic.go:356] "Generic (PLEG): container finished" podID="f4ba7a80-b30b-40b3-a209-674a54c6f096" containerID="9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798" exitCode=0
Oct 30 14:50:16.966226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.966010    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"
Oct 30 14:50:16.966226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.966046    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj" event={"ID":"f4ba7a80-b30b-40b3-a209-674a54c6f096","Type":"ContainerDied","Data":"9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798"}
Oct 30 14:50:16.966226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.966104    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj" event={"ID":"f4ba7a80-b30b-40b3-a209-674a54c6f096","Type":"ContainerDied","Data":"ae594ef12aa0a1588ba74b74d807692ce6682fd22c76be17b72c0a250583e672"}
Oct 30 14:50:16.966226 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.966136    4530 scope.go:122] "RemoveContainer" containerID="9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798"
Oct 30 14:50:16.975714 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.975699    4530 scope.go:122] "RemoveContainer" containerID="9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798"
Oct 30 14:50:16.975980 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:50:16.975964    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798\": container with ID starting with 9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798 not found: ID does not exist" containerID="9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798"
Oct 30 14:50:16.976024 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:16.975988    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798"} err="failed to get container status \"9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798\": rpc error: code = NotFound desc = could not find container \"9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798\": container with ID starting with 9689f5a589ba537303dfb654b3f45819062ec722fb4720a8068ca49c2eeb8798 not found: ID does not exist"
Oct 30 14:50:17.029144 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:17.029118    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"]
Oct 30 14:50:17.046359 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:17.044045    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2df0ab-6bb6d6f87f-4sqjj"]
Oct 30 14:50:17.114055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:17.114010    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" path="/var/lib/kubelet/pods/f2c92412-886d-4c69-9c8e-3cf6fa1a344c/volumes"
Oct 30 14:50:17.114489 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:17.114467    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ba7a80-b30b-40b3-a209-674a54c6f096" path="/var/lib/kubelet/pods/f4ba7a80-b30b-40b3-a209-674a54c6f096/volumes"
Oct 30 14:50:19.507785 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:19.507738    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2" podUID="e54a8e55-53ec-4a30-a4ec-bc39634e5a63" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:50:21.677979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:21.677945    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"]
Oct 30 14:50:21.712478 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:21.712444    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"]
Oct 30 14:50:22.027613 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.027516    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"]
Oct 30 14:50:22.027799 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.027763    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m" podUID="4545aeaf-ba17-4323-9e81-86d345f50d6b" containerName="maas-api" containerID="cri-o://56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b" gracePeriod=30
Oct 30 14:50:22.282227 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.282162    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:50:22.374158 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.374123    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-maas-api-tls\") pod \"4545aeaf-ba17-4323-9e81-86d345f50d6b\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") "
Oct 30 14:50:22.374363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.374232    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-metrics-certs\") pod \"4545aeaf-ba17-4323-9e81-86d345f50d6b\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") "
Oct 30 14:50:22.374363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.374274    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4545aeaf-ba17-4323-9e81-86d345f50d6b-kube-api-access-vkbhm\" (UniqueName: \"kubernetes.io/projected/4545aeaf-ba17-4323-9e81-86d345f50d6b-kube-api-access-vkbhm\") pod \"4545aeaf-ba17-4323-9e81-86d345f50d6b\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") "
Oct 30 14:50:22.374363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.374315    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4545aeaf-ba17-4323-9e81-86d345f50d6b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4545aeaf-ba17-4323-9e81-86d345f50d6b-openshift-service-ca\") pod \"4545aeaf-ba17-4323-9e81-86d345f50d6b\" (UID: \"4545aeaf-ba17-4323-9e81-86d345f50d6b\") "
Oct 30 14:50:22.374868 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.374817    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4545aeaf-ba17-4323-9e81-86d345f50d6b-openshift-service-ca" pod "4545aeaf-ba17-4323-9e81-86d345f50d6b" (UID: "4545aeaf-ba17-4323-9e81-86d345f50d6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:22.376611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.376580    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4545aeaf-ba17-4323-9e81-86d345f50d6b-kube-api-access-vkbhm" pod "4545aeaf-ba17-4323-9e81-86d345f50d6b" (UID: "4545aeaf-ba17-4323-9e81-86d345f50d6b"). InnerVolumeSpecName "kube-api-access-vkbhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:22.376957 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.376934    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-metrics-certs" pod "4545aeaf-ba17-4323-9e81-86d345f50d6b" (UID: "4545aeaf-ba17-4323-9e81-86d345f50d6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:50:22.377092 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.377065    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-maas-api-tls" pod "4545aeaf-ba17-4323-9e81-86d345f50d6b" (UID: "4545aeaf-ba17-4323-9e81-86d345f50d6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:50:22.475515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.475473    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4545aeaf-ba17-4323-9e81-86d345f50d6b-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:22.475515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.475508    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:22.475515 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.475523    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4545aeaf-ba17-4323-9e81-86d345f50d6b-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:22.475804 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.475534    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkbhm\" (UniqueName: \"kubernetes.io/projected/4545aeaf-ba17-4323-9e81-86d345f50d6b-kube-api-access-vkbhm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:22.485017 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.484986    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"]
Oct 30 14:50:22.485385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485372    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f4ba7a80-b30b-40b3-a209-674a54c6f096" containerName="payload-pre-processing"
Oct 30 14:50:22.485433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485385    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ba7a80-b30b-40b3-a209-674a54c6f096" containerName="payload-pre-processing"
Oct 30 14:50:22.485433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485400    4530 cpu_manager.go:414] "containerMap: removing container" podUID="4545aeaf-ba17-4323-9e81-86d345f50d6b" containerName="maas-api"
Oct 30 14:50:22.485433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485406    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4545aeaf-ba17-4323-9e81-86d345f50d6b" containerName="maas-api"
Oct 30 14:50:22.485433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485412    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" containerName="istio-proxy"
Oct 30 14:50:22.485433 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485418    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" containerName="istio-proxy"
Oct 30 14:50:22.485646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485437    4530 cpu_manager.go:414] "containerMap: removing container" podUID="94d586e5-be6a-4560-8e91-3fcf9fd13e04" containerName="revoke-keys"
Oct 30 14:50:22.485646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485441    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d586e5-be6a-4560-8e91-3fcf9fd13e04" containerName="revoke-keys"
Oct 30 14:50:22.485646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485454    4530 cpu_manager.go:414] "containerMap: removing container" podUID="7a8981e5-9b43-41b9-9206-f47a5929d4ff" containerName="revoke-keys"
Oct 30 14:50:22.485646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485459    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8981e5-9b43-41b9-9206-f47a5929d4ff" containerName="revoke-keys"
Oct 30 14:50:22.485646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485522    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ba7a80-b30b-40b3-a209-674a54c6f096" containerName="payload-pre-processing"
Oct 30 14:50:22.485646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485529    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="4545aeaf-ba17-4323-9e81-86d345f50d6b" containerName="maas-api"
Oct 30 14:50:22.485646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485535    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c92412-886d-4c69-9c8e-3cf6fa1a344c" containerName="istio-proxy"
Oct 30 14:50:22.485646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485559    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d586e5-be6a-4560-8e91-3fcf9fd13e04" containerName="revoke-keys"
Oct 30 14:50:22.485646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.485566    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8981e5-9b43-41b9-9206-f47a5929d4ff" containerName="revoke-keys"
Oct 30 14:50:22.488654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.488634    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:22.513898 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.513864    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:50:22.515570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.515502    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"]
Oct 30 14:50:22.525222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.525198    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 14:50:22.576589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.576318    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n8b8\" (UniqueName: \"kubernetes.io/projected/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-kube-api-access-8n8b8\") pod \"maas-api-revoke-sub-593c3d4d-5tvvg\" (UID: \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:22.576589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.576408    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-593c3d4d-5tvvg\" (UID: \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:22.576589 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.576461    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-593c3d4d-5tvvg\" (UID: \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:22.582825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.582789    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"]
Oct 30 14:50:22.587054 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.587028    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:22.604204 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.604162    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"]
Oct 30 14:50:22.683916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.683207    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8b8\" (UniqueName: \"kubernetes.io/projected/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-kube-api-access-8n8b8\") pod \"maas-api-revoke-sub-593c3d4d-5tvvg\" (UID: \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:22.683916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.683309    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-593c3d4d-5tvvg\" (UID: \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:22.683916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.683366    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-593c3d4d-5tvvg\" (UID: \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:22.683916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.683402    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw8s2\" (UniqueName: \"kubernetes.io/projected/4de876db-5913-45bb-988d-1a6617cd6575-kube-api-access-tw8s2\") pod \"maas-api-revoke-sub-bea14b59-qmhvl\" (UID: \"4de876db-5913-45bb-988d-1a6617cd6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:22.683916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.683440    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bea14b59-qmhvl\" (UID: \"4de876db-5913-45bb-988d-1a6617cd6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:22.683916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.683479    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bea14b59-qmhvl\" (UID: \"4de876db-5913-45bb-988d-1a6617cd6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:22.684953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.684660    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-593c3d4d-5tvvg\" (UID: \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:22.688454 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.687759    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-593c3d4d-5tvvg\" (UID: \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:22.710216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.710189    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8b8\" (UniqueName: \"kubernetes.io/projected/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-kube-api-access-8n8b8\") pod \"maas-api-revoke-sub-593c3d4d-5tvvg\" (UID: \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:22.784321 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.784288    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw8s2\" (UniqueName: \"kubernetes.io/projected/4de876db-5913-45bb-988d-1a6617cd6575-kube-api-access-tw8s2\") pod \"maas-api-revoke-sub-bea14b59-qmhvl\" (UID: \"4de876db-5913-45bb-988d-1a6617cd6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:22.784530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.784337    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bea14b59-qmhvl\" (UID: \"4de876db-5913-45bb-988d-1a6617cd6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:22.784530 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.784373    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bea14b59-qmhvl\" (UID: \"4de876db-5913-45bb-988d-1a6617cd6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:22.785838 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.785807    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bea14b59-qmhvl\" (UID: \"4de876db-5913-45bb-988d-1a6617cd6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:22.787260 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.787238    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bea14b59-qmhvl\" (UID: \"4de876db-5913-45bb-988d-1a6617cd6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:22.801463 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.801436    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:22.811659 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.811636    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw8s2\" (UniqueName: \"kubernetes.io/projected/4de876db-5913-45bb-988d-1a6617cd6575-kube-api-access-tw8s2\") pod \"maas-api-revoke-sub-bea14b59-qmhvl\" (UID: \"4de876db-5913-45bb-988d-1a6617cd6575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:22.900068 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:22.900037    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:23.006302 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:23.006266    4530 generic.go:356] "Generic (PLEG): container finished" podID="4545aeaf-ba17-4323-9e81-86d345f50d6b" containerID="56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b" exitCode=0
Oct 30 14:50:23.006464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:23.006345    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"
Oct 30 14:50:23.006464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:23.006351    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m" event={"ID":"4545aeaf-ba17-4323-9e81-86d345f50d6b","Type":"ContainerDied","Data":"56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b"}
Oct 30 14:50:23.006464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:23.006401    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m" event={"ID":"4545aeaf-ba17-4323-9e81-86d345f50d6b","Type":"ContainerDied","Data":"e9d74b1e5685fb8bbbe9b78fd8442ded3cd33040f3539e7355c3cced7246f972"}
Oct 30 14:50:23.006464 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:23.006426    4530 scope.go:122] "RemoveContainer" containerID="56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b"
Oct 30 14:50:23.015898 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:23.015879    4530 scope.go:122] "RemoveContainer" containerID="56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b"
Oct 30 14:50:23.016165 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:50:23.016146    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b\": container with ID starting with 56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b not found: ID does not exist" containerID="56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b"
Oct 30 14:50:23.016242 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:23.016177    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b"} err="failed to get container status \"56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b\": rpc error: code = NotFound desc = could not find container \"56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b\": container with ID starting with 56f2dfae284f5c8827c0ae78416f78a02273e4bf456971d72caa85cda045982b not found: ID does not exist"
Oct 30 14:50:23.067334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:23.067300    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"]
Oct 30 14:50:23.077136 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:23.077109    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-46b13b-64fd48696b-ks25m"]
Oct 30 14:50:23.103762 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:23.103699    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"]
Oct 30 14:50:23.103762 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:50:23.103712    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e610c9a_c5c7_426f_b543_e8c45cf56f8e.slice/crio-f84f79dfb52edc8fb75f7a83eaf8499a8aae22424f43b0c993a14432e564cff6 WatchSource:0}: Error finding container f84f79dfb52edc8fb75f7a83eaf8499a8aae22424f43b0c993a14432e564cff6: Status 404 returned error can't find the container with id f84f79dfb52edc8fb75f7a83eaf8499a8aae22424f43b0c993a14432e564cff6
Oct 30 14:50:23.113623 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:23.113596    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4545aeaf-ba17-4323-9e81-86d345f50d6b" path="/var/lib/kubelet/pods/4545aeaf-ba17-4323-9e81-86d345f50d6b/volumes"
Oct 30 14:50:23.203478 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:50:23.203449    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de876db_5913_45bb_988d_1a6617cd6575.slice/crio-85d1ba314837759adce97c4c95fb8bd7d43a15afad5277d4194f05faa193f119 WatchSource:0}: Error finding container 85d1ba314837759adce97c4c95fb8bd7d43a15afad5277d4194f05faa193f119: Status 404 returned error can't find the container with id 85d1ba314837759adce97c4c95fb8bd7d43a15afad5277d4194f05faa193f119
Oct 30 14:50:23.204194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:23.204166    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"]
Oct 30 14:50:24.014683 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:24.014649    4530 generic.go:356] "Generic (PLEG): container finished" podID="1e610c9a-c5c7-426f-b543-e8c45cf56f8e" containerID="6bd8cb76760082b87af695c128f7c3ffa40b6af783f85adcd8e944ca7170b43c" exitCode=0
Oct 30 14:50:24.015120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:24.014726    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg" event={"ID":"1e610c9a-c5c7-426f-b543-e8c45cf56f8e","Type":"ContainerDied","Data":"6bd8cb76760082b87af695c128f7c3ffa40b6af783f85adcd8e944ca7170b43c"}
Oct 30 14:50:24.015120 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:24.014781    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg" event={"ID":"1e610c9a-c5c7-426f-b543-e8c45cf56f8e","Type":"ContainerStarted","Data":"f84f79dfb52edc8fb75f7a83eaf8499a8aae22424f43b0c993a14432e564cff6"}
Oct 30 14:50:24.016304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:24.016274    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl" event={"ID":"4de876db-5913-45bb-988d-1a6617cd6575","Type":"ContainerStarted","Data":"4839b1a8463fd122e3dbaab55164130e3c9cb7f17853460584a86028a6fcea06"}
Oct 30 14:50:24.016422 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:24.016331    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl" event={"ID":"4de876db-5913-45bb-988d-1a6617cd6575","Type":"ContainerStarted","Data":"85d1ba314837759adce97c4c95fb8bd7d43a15afad5277d4194f05faa193f119"}
Oct 30 14:50:24.108075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:24.108025    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl" podStartSLOduration=2.108011857 podStartE2EDuration="2.108011857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:50:24.106651776 +0000 UTC m=+3689.138550717" watchObservedRunningTime="2026-10-30 14:50:24.108011857 +0000 UTC m=+3689.139910810"
Oct 30 14:50:25.024938 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.024903    4530 generic.go:356] "Generic (PLEG): container finished" podID="4de876db-5913-45bb-988d-1a6617cd6575" containerID="4839b1a8463fd122e3dbaab55164130e3c9cb7f17853460584a86028a6fcea06" exitCode=0
Oct 30 14:50:25.025394 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.024978    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl" event={"ID":"4de876db-5913-45bb-988d-1a6617cd6575","Type":"ContainerDied","Data":"4839b1a8463fd122e3dbaab55164130e3c9cb7f17853460584a86028a6fcea06"}
Oct 30 14:50:25.077304 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.077268    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:50:25.172165 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.172137    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:25.309579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.309527    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-xks-ca\") pod \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\" (UID: \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\") "
Oct 30 14:50:25.309722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.309588    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-kube-api-access-8n8b8\" (UniqueName: \"kubernetes.io/projected/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-kube-api-access-8n8b8\") pod \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\" (UID: \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\") "
Oct 30 14:50:25.309722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.309612    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-openshift-ca\") pod \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\" (UID: \"1e610c9a-c5c7-426f-b543-e8c45cf56f8e\") "
Oct 30 14:50:25.310031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.310007    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-openshift-ca" pod "1e610c9a-c5c7-426f-b543-e8c45cf56f8e" (UID: "1e610c9a-c5c7-426f-b543-e8c45cf56f8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:25.311797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.311769    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-kube-api-access-8n8b8" pod "1e610c9a-c5c7-426f-b543-e8c45cf56f8e" (UID: "1e610c9a-c5c7-426f-b543-e8c45cf56f8e"). InnerVolumeSpecName "kube-api-access-8n8b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:25.311920 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.311772    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-xks-ca" pod "1e610c9a-c5c7-426f-b543-e8c45cf56f8e" (UID: "1e610c9a-c5c7-426f-b543-e8c45cf56f8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:50:25.410729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.410700    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:25.410729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.410725    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n8b8\" (UniqueName: \"kubernetes.io/projected/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-kube-api-access-8n8b8\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:25.410729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:25.410735    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e610c9a-c5c7-426f-b543-e8c45cf56f8e-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:26.031925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.031891    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"
Oct 30 14:50:26.032314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.031920    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg" event={"ID":"1e610c9a-c5c7-426f-b543-e8c45cf56f8e","Type":"ContainerDied","Data":"f84f79dfb52edc8fb75f7a83eaf8499a8aae22424f43b0c993a14432e564cff6"}
Oct 30 14:50:26.032314 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.031976    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f84f79dfb52edc8fb75f7a83eaf8499a8aae22424f43b0c993a14432e564cff6"
Oct 30 14:50:26.173350 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.173320    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:26.318354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.318316    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-openshift-ca\") pod \"4de876db-5913-45bb-988d-1a6617cd6575\" (UID: \"4de876db-5913-45bb-988d-1a6617cd6575\") "
Oct 30 14:50:26.318611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.318386    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de876db-5913-45bb-988d-1a6617cd6575-kube-api-access-tw8s2\" (UniqueName: \"kubernetes.io/projected/4de876db-5913-45bb-988d-1a6617cd6575-kube-api-access-tw8s2\") pod \"4de876db-5913-45bb-988d-1a6617cd6575\" (UID: \"4de876db-5913-45bb-988d-1a6617cd6575\") "
Oct 30 14:50:26.318611 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.318456    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-xks-ca\") pod \"4de876db-5913-45bb-988d-1a6617cd6575\" (UID: \"4de876db-5913-45bb-988d-1a6617cd6575\") "
Oct 30 14:50:26.318792 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.318765    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-openshift-ca" pod "4de876db-5913-45bb-988d-1a6617cd6575" (UID: "4de876db-5913-45bb-988d-1a6617cd6575"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:26.320574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.320529    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de876db-5913-45bb-988d-1a6617cd6575-kube-api-access-tw8s2" pod "4de876db-5913-45bb-988d-1a6617cd6575" (UID: "4de876db-5913-45bb-988d-1a6617cd6575"). InnerVolumeSpecName "kube-api-access-tw8s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:26.320574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.320565    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-xks-ca" pod "4de876db-5913-45bb-988d-1a6617cd6575" (UID: "4de876db-5913-45bb-988d-1a6617cd6575"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:50:26.420222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.420170    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:26.420222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.420216    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw8s2\" (UniqueName: \"kubernetes.io/projected/4de876db-5913-45bb-988d-1a6617cd6575-kube-api-access-tw8s2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:26.420222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.420228    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de876db-5913-45bb-988d-1a6617cd6575-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:26.678595 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.678467    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp" podUID="ef6239cf-ec71-4299-8b6d-ef31d72c3162" containerName="payload-processing" containerID="cri-o://7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40" gracePeriod=25
Oct 30 14:50:26.713043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.712995    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z" podUID="da680b84-e339-4baf-adfb-e6519af3fbbd" containerName="payload-pre-processing" containerID="cri-o://05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597" gracePeriod=25
Oct 30 14:50:26.952998 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.952969    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"
Oct 30 14:50:26.978951 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:26.978925    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"
Oct 30 14:50:27.026913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.026885    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6239cf-ec71-4299-8b6d-ef31d72c3162-kube-api-access-rcpx2\" (UniqueName: \"kubernetes.io/projected/ef6239cf-ec71-4299-8b6d-ef31d72c3162-kube-api-access-rcpx2\") pod \"ef6239cf-ec71-4299-8b6d-ef31d72c3162\" (UID: \"ef6239cf-ec71-4299-8b6d-ef31d72c3162\") "
Oct 30 14:50:27.027075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.026944    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6239cf-ec71-4299-8b6d-ef31d72c3162-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6239cf-ec71-4299-8b6d-ef31d72c3162-plugins-config-volume\") pod \"ef6239cf-ec71-4299-8b6d-ef31d72c3162\" (UID: \"ef6239cf-ec71-4299-8b6d-ef31d72c3162\") "
Oct 30 14:50:27.027332 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.027308    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6239cf-ec71-4299-8b6d-ef31d72c3162-plugins-config-volume" pod "ef6239cf-ec71-4299-8b6d-ef31d72c3162" (UID: "ef6239cf-ec71-4299-8b6d-ef31d72c3162"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:27.029076 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.029054    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6239cf-ec71-4299-8b6d-ef31d72c3162-kube-api-access-rcpx2" pod "ef6239cf-ec71-4299-8b6d-ef31d72c3162" (UID: "ef6239cf-ec71-4299-8b6d-ef31d72c3162"). InnerVolumeSpecName "kube-api-access-rcpx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:27.038140 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.038113    4530 generic.go:356] "Generic (PLEG): container finished" podID="ef6239cf-ec71-4299-8b6d-ef31d72c3162" containerID="7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40" exitCode=0
Oct 30 14:50:27.038474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.038158    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"
Oct 30 14:50:27.038474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.038197    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp" event={"ID":"ef6239cf-ec71-4299-8b6d-ef31d72c3162","Type":"ContainerDied","Data":"7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40"}
Oct 30 14:50:27.038474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.038280    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp" event={"ID":"ef6239cf-ec71-4299-8b6d-ef31d72c3162","Type":"ContainerDied","Data":"aace5e15d468166e31135e936776cd350e21cdda83f91440982e12e858c630b6"}
Oct 30 14:50:27.038474 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.038307    4530 scope.go:122] "RemoveContainer" containerID="7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40"
Oct 30 14:50:27.039631 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.039613    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"
Oct 30 14:50:27.039718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.039654    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl" event={"ID":"4de876db-5913-45bb-988d-1a6617cd6575","Type":"ContainerDied","Data":"85d1ba314837759adce97c4c95fb8bd7d43a15afad5277d4194f05faa193f119"}
Oct 30 14:50:27.039718 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.039705    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85d1ba314837759adce97c4c95fb8bd7d43a15afad5277d4194f05faa193f119"
Oct 30 14:50:27.041085 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.041061    4530 generic.go:356] "Generic (PLEG): container finished" podID="da680b84-e339-4baf-adfb-e6519af3fbbd" containerID="05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597" exitCode=0
Oct 30 14:50:27.041192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.041099    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z" event={"ID":"da680b84-e339-4baf-adfb-e6519af3fbbd","Type":"ContainerDied","Data":"05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597"}
Oct 30 14:50:27.041192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.041112    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"
Oct 30 14:50:27.041192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.041134    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z" event={"ID":"da680b84-e339-4baf-adfb-e6519af3fbbd","Type":"ContainerDied","Data":"a4835b82c64e830d8e0e317999e63f24714159cc4da7df25058d24807adb78ed"}
Oct 30 14:50:27.051670 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.051652    4530 scope.go:122] "RemoveContainer" containerID="7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40"
Oct 30 14:50:27.052015 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:50:27.051963    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40\": container with ID starting with 7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40 not found: ID does not exist" containerID="7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40"
Oct 30 14:50:27.052015 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.051994    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40"} err="failed to get container status \"7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40\": rpc error: code = NotFound desc = could not find container \"7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40\": container with ID starting with 7e2bff9358b0da94dd94c17922939011377055cc881385ea68c2c7dfd0f70d40 not found: ID does not exist"
Oct 30 14:50:27.052152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.052018    4530 scope.go:122] "RemoveContainer" containerID="05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597"
Oct 30 14:50:27.060654 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.060633    4530 scope.go:122] "RemoveContainer" containerID="05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597"
Oct 30 14:50:27.060973 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:50:27.060953    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597\": container with ID starting with 05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597 not found: ID does not exist" containerID="05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597"
Oct 30 14:50:27.061026 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.060978    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597"} err="failed to get container status \"05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597\": rpc error: code = NotFound desc = could not find container \"05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597\": container with ID starting with 05b5b48d2fb221b22413345c165627980b6d928568b48db21a3bf28c31f1f597 not found: ID does not exist"
Oct 30 14:50:27.128295 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.128266    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da680b84-e339-4baf-adfb-e6519af3fbbd-kube-api-access-6m9lv\" (UniqueName: \"kubernetes.io/projected/da680b84-e339-4baf-adfb-e6519af3fbbd-kube-api-access-6m9lv\") pod \"da680b84-e339-4baf-adfb-e6519af3fbbd\" (UID: \"da680b84-e339-4baf-adfb-e6519af3fbbd\") "
Oct 30 14:50:27.128451 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.128336    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da680b84-e339-4baf-adfb-e6519af3fbbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da680b84-e339-4baf-adfb-e6519af3fbbd-plugins-config-volume\") pod \"da680b84-e339-4baf-adfb-e6519af3fbbd\" (UID: \"da680b84-e339-4baf-adfb-e6519af3fbbd\") "
Oct 30 14:50:27.128717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.128695    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcpx2\" (UniqueName: \"kubernetes.io/projected/ef6239cf-ec71-4299-8b6d-ef31d72c3162-kube-api-access-rcpx2\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:27.128816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.128722    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6239cf-ec71-4299-8b6d-ef31d72c3162-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:27.128816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.128726    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da680b84-e339-4baf-adfb-e6519af3fbbd-plugins-config-volume" pod "da680b84-e339-4baf-adfb-e6519af3fbbd" (UID: "da680b84-e339-4baf-adfb-e6519af3fbbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:27.130733 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.130712    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da680b84-e339-4baf-adfb-e6519af3fbbd-kube-api-access-6m9lv" pod "da680b84-e339-4baf-adfb-e6519af3fbbd" (UID: "da680b84-e339-4baf-adfb-e6519af3fbbd"). InnerVolumeSpecName "kube-api-access-6m9lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:27.143065 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.143036    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"]
Oct 30 14:50:27.148649 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.148628    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-00926c-88774844d-mz5bp"]
Oct 30 14:50:27.229368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.229311    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da680b84-e339-4baf-adfb-e6519af3fbbd-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:27.229368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.229334    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m9lv\" (UniqueName: \"kubernetes.io/projected/da680b84-e339-4baf-adfb-e6519af3fbbd-kube-api-access-6m9lv\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:27.394704 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.394673    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"]
Oct 30 14:50:27.400203 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:27.400174    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-00926c-77f9968595-6zj5z"]
Oct 30 14:50:29.113513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:29.113476    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da680b84-e339-4baf-adfb-e6519af3fbbd" path="/var/lib/kubelet/pods/da680b84-e339-4baf-adfb-e6519af3fbbd/volumes"
Oct 30 14:50:29.114083 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:29.114059    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6239cf-ec71-4299-8b6d-ef31d72c3162" path="/var/lib/kubelet/pods/ef6239cf-ec71-4299-8b6d-ef31d72c3162/volumes"
Oct 30 14:50:29.507443 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:29.507351    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2" podUID="e54a8e55-53ec-4a30-a4ec-bc39634e5a63" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:50:33.268646 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:33.268619    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"
Oct 30 14:50:33.383775 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:33.383743    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-kube-api-access-ndbc5\" (UniqueName: \"kubernetes.io/projected/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-kube-api-access-ndbc5\") pod \"e54a8e55-53ec-4a30-a4ec-bc39634e5a63\" (UID: \"e54a8e55-53ec-4a30-a4ec-bc39634e5a63\") "
Oct 30 14:50:33.383946 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:33.383832    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-plugins-config-volume\") pod \"e54a8e55-53ec-4a30-a4ec-bc39634e5a63\" (UID: \"e54a8e55-53ec-4a30-a4ec-bc39634e5a63\") "
Oct 30 14:50:33.384198 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:33.384175    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-plugins-config-volume" pod "e54a8e55-53ec-4a30-a4ec-bc39634e5a63" (UID: "e54a8e55-53ec-4a30-a4ec-bc39634e5a63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:50:33.385984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:33.385958    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-kube-api-access-ndbc5" pod "e54a8e55-53ec-4a30-a4ec-bc39634e5a63" (UID: "e54a8e55-53ec-4a30-a4ec-bc39634e5a63"). InnerVolumeSpecName "kube-api-access-ndbc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:50:33.485354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:33.485314    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndbc5\" (UniqueName: \"kubernetes.io/projected/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-kube-api-access-ndbc5\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:33.485354 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:33.485348    4530 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54a8e55-53ec-4a30-a4ec-bc39634e5a63-plugins-config-volume\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:50:34.084810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:34.084769    4530 generic.go:356] "Generic (PLEG): container finished" podID="e54a8e55-53ec-4a30-a4ec-bc39634e5a63" containerID="ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2" exitCode=0
Oct 30 14:50:34.084810 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:34.084811    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"
Oct 30 14:50:34.085075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:34.084851    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2" event={"ID":"e54a8e55-53ec-4a30-a4ec-bc39634e5a63","Type":"ContainerDied","Data":"ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2"}
Oct 30 14:50:34.085075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:34.084910    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2" event={"ID":"e54a8e55-53ec-4a30-a4ec-bc39634e5a63","Type":"ContainerDied","Data":"138f2fdda6b8899f5619d906033e233ee2b68ba49f6601a4c7be2226c8e64c56"}
Oct 30 14:50:34.085075 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:34.084936    4530 scope.go:122] "RemoveContainer" containerID="ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2"
Oct 30 14:50:34.094585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:34.094568    4530 scope.go:122] "RemoveContainer" containerID="ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2"
Oct 30 14:50:34.094861 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:50:34.094838    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2\": container with ID starting with ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2 not found: ID does not exist" containerID="ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2"
Oct 30 14:50:34.094938 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:34.094874    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2"} err="failed to get container status \"ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2\": rpc error: code = NotFound desc = could not find container \"ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2\": container with ID starting with ceb0c9cba1875c3cd16a8802ffb24d027d035efc944be868fce9a0c45a0d05e2 not found: ID does not exist"
Oct 30 14:50:34.144861 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:34.144837    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"]
Oct 30 14:50:34.159383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:34.159354    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-46b13b-646fb546c8-4xsd2"]
Oct 30 14:50:35.114635 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:35.114593    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54a8e55-53ec-4a30-a4ec-bc39634e5a63" path="/var/lib/kubelet/pods/e54a8e55-53ec-4a30-a4ec-bc39634e5a63/volumes"
Oct 30 14:50:36.245648 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:36.245613    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"]
Oct 30 14:50:36.259507 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:36.259478    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-39529e-zqxcp"]
Oct 30 14:50:37.112745 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:37.112713    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a8981e5-9b43-41b9-9206-f47a5929d4ff" path="/var/lib/kubelet/pods/7a8981e5-9b43-41b9-9206-f47a5929d4ff/volumes"
Oct 30 14:50:38.481765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:50:38.481729    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:50:41.366391 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:50:41.366351    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:51:01.425881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.425840    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75db85b96b-jqftl"]
Oct 30 14:51:01.426449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426426    4530 cpu_manager.go:414] "containerMap: removing container" podUID="da680b84-e339-4baf-adfb-e6519af3fbbd" containerName="payload-pre-processing"
Oct 30 14:51:01.426508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426454    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da680b84-e339-4baf-adfb-e6519af3fbbd" containerName="payload-pre-processing"
Oct 30 14:51:01.426508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426475    4530 cpu_manager.go:414] "containerMap: removing container" podUID="1e610c9a-c5c7-426f-b543-e8c45cf56f8e" containerName="revoke-keys"
Oct 30 14:51:01.426508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426484    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e610c9a-c5c7-426f-b543-e8c45cf56f8e" containerName="revoke-keys"
Oct 30 14:51:01.426508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426496    4530 cpu_manager.go:414] "containerMap: removing container" podUID="e54a8e55-53ec-4a30-a4ec-bc39634e5a63" containerName="payload-processing"
Oct 30 14:51:01.426508 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426506    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54a8e55-53ec-4a30-a4ec-bc39634e5a63" containerName="payload-processing"
Oct 30 14:51:01.426721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426523    4530 cpu_manager.go:414] "containerMap: removing container" podUID="ef6239cf-ec71-4299-8b6d-ef31d72c3162" containerName="payload-processing"
Oct 30 14:51:01.426721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426531    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6239cf-ec71-4299-8b6d-ef31d72c3162" containerName="payload-processing"
Oct 30 14:51:01.426721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426568    4530 cpu_manager.go:414] "containerMap: removing container" podUID="4de876db-5913-45bb-988d-1a6617cd6575" containerName="revoke-keys"
Oct 30 14:51:01.426721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426575    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de876db-5913-45bb-988d-1a6617cd6575" containerName="revoke-keys"
Oct 30 14:51:01.426721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426679    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de876db-5913-45bb-988d-1a6617cd6575" containerName="revoke-keys"
Oct 30 14:51:01.426721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426702    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="da680b84-e339-4baf-adfb-e6519af3fbbd" containerName="payload-pre-processing"
Oct 30 14:51:01.426721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426713    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e610c9a-c5c7-426f-b543-e8c45cf56f8e" containerName="revoke-keys"
Oct 30 14:51:01.426721 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426722    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54a8e55-53ec-4a30-a4ec-bc39634e5a63" containerName="payload-processing"
Oct 30 14:51:01.427011 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.426733    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6239cf-ec71-4299-8b6d-ef31d72c3162" containerName="payload-processing"
Oct 30 14:51:01.430421 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.430403    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:01.462310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.462278    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-666kd\""
Oct 30 14:51:01.462310 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.462301    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 14:51:01.462577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.462324    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 14:51:01.462577 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.462352    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 14:51:01.464953 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.464927    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75db85b96b-jqftl"]
Oct 30 14:51:01.532123 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.532088    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-metrics-certs\") pod \"maas-controller-75db85b96b-jqftl\" (UID: \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\") " pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:01.532282 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.532242    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-webhook-cert\") pod \"maas-controller-75db85b96b-jqftl\" (UID: \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\") " pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:01.532343 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.532281    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9mbl\" (UniqueName: \"kubernetes.io/projected/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-kube-api-access-g9mbl\") pod \"maas-controller-75db85b96b-jqftl\" (UID: \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\") " pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:01.633169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.633132    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-webhook-cert\") pod \"maas-controller-75db85b96b-jqftl\" (UID: \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\") " pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:01.633169 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.633173    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mbl\" (UniqueName: \"kubernetes.io/projected/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-kube-api-access-g9mbl\") pod \"maas-controller-75db85b96b-jqftl\" (UID: \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\") " pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:01.633363 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.633199    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-metrics-certs\") pod \"maas-controller-75db85b96b-jqftl\" (UID: \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\") " pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:01.635805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.635776    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-metrics-certs\") pod \"maas-controller-75db85b96b-jqftl\" (UID: \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\") " pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:01.635805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.635793    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-webhook-cert\") pod \"maas-controller-75db85b96b-jqftl\" (UID: \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\") " pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:01.659777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.659750    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9mbl\" (UniqueName: \"kubernetes.io/projected/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-kube-api-access-g9mbl\") pod \"maas-controller-75db85b96b-jqftl\" (UID: \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\") " pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:01.744764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:01.744680    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:02.049334 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:02.049300    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75db85b96b-jqftl"]
Oct 30 14:51:02.049532 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:51:02.049511    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce729ee_bcfb_48d3_86f7_02bf863a1ec2.slice/crio-7b9c493344f11a5d3af7a783f17a78bc006fbfd93d0e014187cc7d97ba8d7228 WatchSource:0}: Error finding container 7b9c493344f11a5d3af7a783f17a78bc006fbfd93d0e014187cc7d97ba8d7228: Status 404 returned error can't find the container with id 7b9c493344f11a5d3af7a783f17a78bc006fbfd93d0e014187cc7d97ba8d7228
Oct 30 14:51:02.050961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:02.050944    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:51:02.256405 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:02.256368    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75db85b96b-jqftl" event={"ID":"cce729ee-bcfb-48d3-86f7-02bf863a1ec2","Type":"ContainerStarted","Data":"7b9c493344f11a5d3af7a783f17a78bc006fbfd93d0e014187cc7d97ba8d7228"}
Oct 30 14:51:04.269995 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:04.269895    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75db85b96b-jqftl" event={"ID":"cce729ee-bcfb-48d3-86f7-02bf863a1ec2","Type":"ContainerStarted","Data":"9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b"}
Oct 30 14:51:04.270387 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:04.270107    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:04.320293 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:04.320245    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75db85b96b-jqftl" podStartSLOduration=1.541928517 podStartE2EDuration="3.320232365s" totalImagesPullingTime="1.778303848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:04.318853677 +0000 UTC m=+3729.350752616" watchObservedRunningTime="2026-10-30 14:51:04.320232365 +0000 UTC m=+3729.352131334"
Oct 30 14:51:11.366843 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:51:11.366747    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:51:11.389574 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.389527    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"]
Oct 30 14:51:11.399069 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.399037    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.407925 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.407897    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"]
Oct 30 14:51:11.411190 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.411173    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.419161 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.419141    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5drtm\""
Oct 30 14:51:11.420033 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.420008    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"]
Oct 30 14:51:11.434529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.434493    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"]
Oct 30 14:51:11.519308 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519266    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.519308 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519313    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4ll5\" (UniqueName: \"kubernetes.io/projected/b9c516e0-9b3d-4398-a08b-c9db1a762db1-kube-api-access-q4ll5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.519608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519413    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.519608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519455    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.519608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519567    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.519758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519621    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.519758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519644    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.519758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519671    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k57gk\" (UniqueName: \"kubernetes.io/projected/075cda4c-17d3-48d0-801c-0a0101cb3a5b-kube-api-access-k57gk\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.519758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519718    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.519758 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519752    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.520041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519810    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.520041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519859    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.520041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519889    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.520041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519961    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.520041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.519993    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.520041 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.520027    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.520258 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.520064    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.520258 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.520080    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.621018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.620934    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.621018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.620977    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4ll5\" (UniqueName: \"kubernetes.io/projected/b9c516e0-9b3d-4398-a08b-c9db1a762db1-kube-api-access-q4ll5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.621018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621017    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.621306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621038    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.621306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621070    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.621306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621120    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.621306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621148    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.621306 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621174    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k57gk\" (UniqueName: \"kubernetes.io/projected/075cda4c-17d3-48d0-801c-0a0101cb3a5b-kube-api-access-k57gk\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.621629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621339    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.621629 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621483    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.621751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621682    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.621751 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621729    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.621869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621756    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.621869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621814    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.621869 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621844    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.622046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621916    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.622046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621936    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.622046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.621971    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.622046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.622003    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.622046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.622014    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.622046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.622032    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.622373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.622047    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.622373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.622060    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.622373 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.622304    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.622570 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.622425    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.622643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.622603    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.622643 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.622625    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.622766 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.622722    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.623977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.623953    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.624104 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.624081    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.624247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.624228    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.624784 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.624759    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.648110 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.648078    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4ll5\" (UniqueName: \"kubernetes.io/projected/b9c516e0-9b3d-4398-a08b-c9db1a762db1-kube-api-access-q4ll5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.648241 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.648204    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/075cda4c-17d3-48d0-801c-0a0101cb3a5b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.648309 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.648280    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k57gk\" (UniqueName: \"kubernetes.io/projected/075cda4c-17d3-48d0-801c-0a0101cb3a5b-kube-api-access-k57gk\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-52g42\" (UID: \"075cda4c-17d3-48d0-801c-0a0101cb3a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:11.648532 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.648514    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c516e0-9b3d-4398-a08b-c9db1a762db1-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d9pw9\" (UID: \"b9c516e0-9b3d-4398-a08b-c9db1a762db1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.712934 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.712906    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:11.725811 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:11.725777    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:12.045274 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:51:12.045233    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075cda4c_17d3_48d0_801c_0a0101cb3a5b.slice/crio-4bab64ceb984df5027fd4a64458082d1082e3c0878121fb1c2aaad7ff6bf31a6 WatchSource:0}: Error finding container 4bab64ceb984df5027fd4a64458082d1082e3c0878121fb1c2aaad7ff6bf31a6: Status 404 returned error can't find the container with id 4bab64ceb984df5027fd4a64458082d1082e3c0878121fb1c2aaad7ff6bf31a6
Oct 30 14:51:12.045879 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:51:12.045846    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c516e0_9b3d_4398_a08b_c9db1a762db1.slice/crio-f9af9d79da721cea5c8c35e14b05675a2f81a83e19c4acba153e32c12c521059 WatchSource:0}: Error finding container f9af9d79da721cea5c8c35e14b05675a2f81a83e19c4acba153e32c12c521059: Status 404 returned error can't find the container with id f9af9d79da721cea5c8c35e14b05675a2f81a83e19c4acba153e32c12c521059
Oct 30 14:51:12.045879 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.045858    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"]
Oct 30 14:51:12.047748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.047707    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:51:12.047833 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.047796    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:51:12.047893 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.047846    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:51:12.048222 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.048196    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:51:12.048301 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.048257    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:51:12.048429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.048296    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:51:12.049070 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.049045    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"]
Oct 30 14:51:12.316853 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.316813    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42" event={"ID":"075cda4c-17d3-48d0-801c-0a0101cb3a5b","Type":"ContainerStarted","Data":"7d298dbcc9aa2d08b7f005419a49898cf41b36eec427bab9c9d1b1d3c24c05f5"}
Oct 30 14:51:12.317051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.316885    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42" event={"ID":"075cda4c-17d3-48d0-801c-0a0101cb3a5b","Type":"ContainerStarted","Data":"4bab64ceb984df5027fd4a64458082d1082e3c0878121fb1c2aaad7ff6bf31a6"}
Oct 30 14:51:12.318712 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.318679    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9" event={"ID":"b9c516e0-9b3d-4398-a08b-c9db1a762db1","Type":"ContainerStarted","Data":"f7395b1e68cf536ec52b86e9da90c99fea2135311916d0f98ceec6cc1d442fba"}
Oct 30 14:51:12.318867 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.318744    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9" event={"ID":"b9c516e0-9b3d-4398-a08b-c9db1a762db1","Type":"ContainerStarted","Data":"f9af9d79da721cea5c8c35e14b05675a2f81a83e19c4acba153e32c12c521059"}
Oct 30 14:51:12.370175 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.370113    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42" podStartSLOduration=1.370100218 podStartE2EDuration="1.370100218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:12.367835803 +0000 UTC m=+3737.399734754" watchObservedRunningTime="2026-10-30 14:51:12.370100218 +0000 UTC m=+3737.401999157"
Oct 30 14:51:12.421979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.421912    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9" podStartSLOduration=1.421892777 podStartE2EDuration="1.421892777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:12.41833364 +0000 UTC m=+3737.450232581" watchObservedRunningTime="2026-10-30 14:51:12.421892777 +0000 UTC m=+3737.453791718"
Oct 30 14:51:12.717632 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.713642    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:12.729678 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:12.729630    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:13.717566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:13.716615    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 14:51:13.717566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:13.716679    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9" podUID="b9c516e0-9b3d-4398-a08b-c9db1a762db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 14:51:13.730082 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:13.730052    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-52g42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 30 14:51:13.730294 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:13.730271    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42" podUID="075cda4c-17d3-48d0-801c-0a0101cb3a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 30 14:51:14.720055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:14.720007    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 14:51:14.721076 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:14.720645    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9" podUID="b9c516e0-9b3d-4398-a08b-c9db1a762db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 14:51:14.726950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:14.726919    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-52g42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 30 14:51:14.727174 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:14.727146    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42" podUID="075cda4c-17d3-48d0-801c-0a0101cb3a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 30 14:51:15.277753 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:15.277717    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:15.717117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:15.717065    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 14:51:15.717445 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:15.717415    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9" podUID="b9c516e0-9b3d-4398-a08b-c9db1a762db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 14:51:15.729926 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:15.729894    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-52g42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 30 14:51:15.730361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:15.730336    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42" podUID="075cda4c-17d3-48d0-801c-0a0101cb3a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 30 14:51:16.717566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:16.716995    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 14:51:16.717566 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:16.717065    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9" podUID="b9c516e0-9b3d-4398-a08b-c9db1a762db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 14:51:16.726812 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:16.726776    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-52g42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 30 14:51:16.727119 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:16.727076    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42" podUID="075cda4c-17d3-48d0-801c-0a0101cb3a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 30 14:51:17.716565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:17.716088    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 14:51:17.716565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:17.716171    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9" podUID="b9c516e0-9b3d-4398-a08b-c9db1a762db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 14:51:17.730021 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:17.729970    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-52g42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 30 14:51:17.730231 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:17.730039    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42" podUID="075cda4c-17d3-48d0-801c-0a0101cb3a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 30 14:51:18.713666 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:18.713622    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 14:51:18.713889 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:18.713693    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9" podUID="b9c516e0-9b3d-4398-a08b-c9db1a762db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 14:51:18.727133 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:18.727093    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-52g42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 30 14:51:18.727608 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:18.727147    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42" podUID="075cda4c-17d3-48d0-801c-0a0101cb3a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 30 14:51:20.431152 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:20.431114    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:20.432450 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:20.432424    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:20.433056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:20.433032    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42"
Oct 30 14:51:20.455917 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:20.455886    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:20.456624 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:20.456602    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:20.457418 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:20.457395    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d9pw9"
Oct 30 14:51:22.504748 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:22.504677    4530 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" podUID="e329b8d309e1f8141c44f91d4bb26de8" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 14:51:41.366799 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:51:41.366761    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:51:46.436676 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:46.436640    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75db85b96b-jqftl"]
Oct 30 14:51:46.437116 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:46.436851    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-75db85b96b-jqftl" podUID="cce729ee-bcfb-48d3-86f7-02bf863a1ec2" containerName="manager" containerID="cri-o://9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b" gracePeriod=30
Oct 30 14:51:46.690097 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:46.690035    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:46.777385 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:46.777350    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-metrics-certs\") pod \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\" (UID: \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\") "
Oct 30 14:51:46.777591 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:46.777396    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-webhook-cert\" (UniqueName: \"kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-webhook-cert\") pod \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\" (UID: \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\") "
Oct 30 14:51:46.777591 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:46.777521    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-kube-api-access-g9mbl\" (UniqueName: \"kubernetes.io/projected/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-kube-api-access-g9mbl\") pod \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\" (UID: \"cce729ee-bcfb-48d3-86f7-02bf863a1ec2\") "
Oct 30 14:51:46.779586 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:46.779527    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-metrics-certs" pod "cce729ee-bcfb-48d3-86f7-02bf863a1ec2" (UID: "cce729ee-bcfb-48d3-86f7-02bf863a1ec2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:51:46.779738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:46.779693    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-webhook-cert" pod "cce729ee-bcfb-48d3-86f7-02bf863a1ec2" (UID: "cce729ee-bcfb-48d3-86f7-02bf863a1ec2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:51:46.779738 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:46.779719    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-kube-api-access-g9mbl" pod "cce729ee-bcfb-48d3-86f7-02bf863a1ec2" (UID: "cce729ee-bcfb-48d3-86f7-02bf863a1ec2"). InnerVolumeSpecName "kube-api-access-g9mbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:51:46.878970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:46.878935    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:51:46.878970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:46.878964    4530 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-webhook-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:51:46.878970 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:46.878975    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9mbl\" (UniqueName: \"kubernetes.io/projected/cce729ee-bcfb-48d3-86f7-02bf863a1ec2-kube-api-access-g9mbl\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:51:47.550202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:47.550169    4530 generic.go:356] "Generic (PLEG): container finished" podID="cce729ee-bcfb-48d3-86f7-02bf863a1ec2" containerID="9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b" exitCode=0
Oct 30 14:51:47.550661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:47.550217    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75db85b96b-jqftl"
Oct 30 14:51:47.550661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:47.550263    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75db85b96b-jqftl" event={"ID":"cce729ee-bcfb-48d3-86f7-02bf863a1ec2","Type":"ContainerDied","Data":"9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b"}
Oct 30 14:51:47.550661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:47.550314    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75db85b96b-jqftl" event={"ID":"cce729ee-bcfb-48d3-86f7-02bf863a1ec2","Type":"ContainerDied","Data":"7b9c493344f11a5d3af7a783f17a78bc006fbfd93d0e014187cc7d97ba8d7228"}
Oct 30 14:51:47.550661 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:47.550342    4530 scope.go:122] "RemoveContainer" containerID="9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b"
Oct 30 14:51:47.560154 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:47.560133    4530 scope.go:122] "RemoveContainer" containerID="9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b"
Oct 30 14:51:47.560462 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:51:47.560437    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b\": container with ID starting with 9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b not found: ID does not exist" containerID="9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b"
Oct 30 14:51:47.560581 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:47.560470    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b"} err="failed to get container status \"9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b\": rpc error: code = NotFound desc = could not find container \"9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b\": container with ID starting with 9e6438488d6423cfb94ad47f7a11f15a2fb73ccd1a8b03854f9707e8c22b625b not found: ID does not exist"
Oct 30 14:51:47.611383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:47.611354    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75db85b96b-jqftl"]
Oct 30 14:51:47.625495 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:47.625467    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75db85b96b-jqftl"]
Oct 30 14:51:48.842221 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.842187    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75db85b96b-t7j2b"]
Oct 30 14:51:48.842622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.842603    4530 cpu_manager.go:414] "containerMap: removing container" podUID="cce729ee-bcfb-48d3-86f7-02bf863a1ec2" containerName="manager"
Oct 30 14:51:48.842622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.842616    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce729ee-bcfb-48d3-86f7-02bf863a1ec2" containerName="manager"
Oct 30 14:51:48.842707 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.842693    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce729ee-bcfb-48d3-86f7-02bf863a1ec2" containerName="manager"
Oct 30 14:51:48.847287 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.847264    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:51:48.880524 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.880486    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 14:51:48.880706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.880536    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 14:51:48.880706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.880560    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-666kd\""
Oct 30 14:51:48.880706 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.880671    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 14:51:48.883265 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.883243    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75db85b96b-t7j2b"]
Oct 30 14:51:48.894961 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.894932    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skrbx\" (UniqueName: \"kubernetes.io/projected/abe7f074-2d3f-44b8-9d3d-e54407143172-kube-api-access-skrbx\") pod \"maas-controller-75db85b96b-t7j2b\" (UID: \"abe7f074-2d3f-44b8-9d3d-e54407143172\") " pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:51:48.895126 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.894986    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe7f074-2d3f-44b8-9d3d-e54407143172-metrics-certs\") pod \"maas-controller-75db85b96b-t7j2b\" (UID: \"abe7f074-2d3f-44b8-9d3d-e54407143172\") " pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:51:48.895228 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.895205    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abe7f074-2d3f-44b8-9d3d-e54407143172-webhook-cert\") pod \"maas-controller-75db85b96b-t7j2b\" (UID: \"abe7f074-2d3f-44b8-9d3d-e54407143172\") " pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:51:48.995820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.995772    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abe7f074-2d3f-44b8-9d3d-e54407143172-webhook-cert\") pod \"maas-controller-75db85b96b-t7j2b\" (UID: \"abe7f074-2d3f-44b8-9d3d-e54407143172\") " pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:51:48.995820 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.995823    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skrbx\" (UniqueName: \"kubernetes.io/projected/abe7f074-2d3f-44b8-9d3d-e54407143172-kube-api-access-skrbx\") pod \"maas-controller-75db85b96b-t7j2b\" (UID: \"abe7f074-2d3f-44b8-9d3d-e54407143172\") " pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:51:48.996112 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.995864    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe7f074-2d3f-44b8-9d3d-e54407143172-metrics-certs\") pod \"maas-controller-75db85b96b-t7j2b\" (UID: \"abe7f074-2d3f-44b8-9d3d-e54407143172\") " pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:51:48.998456 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.998432    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe7f074-2d3f-44b8-9d3d-e54407143172-metrics-certs\") pod \"maas-controller-75db85b96b-t7j2b\" (UID: \"abe7f074-2d3f-44b8-9d3d-e54407143172\") " pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:51:48.998552 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:48.998468    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abe7f074-2d3f-44b8-9d3d-e54407143172-webhook-cert\") pod \"maas-controller-75db85b96b-t7j2b\" (UID: \"abe7f074-2d3f-44b8-9d3d-e54407143172\") " pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:51:49.022763 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:49.022738    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skrbx\" (UniqueName: \"kubernetes.io/projected/abe7f074-2d3f-44b8-9d3d-e54407143172-kube-api-access-skrbx\") pod \"maas-controller-75db85b96b-t7j2b\" (UID: \"abe7f074-2d3f-44b8-9d3d-e54407143172\") " pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:51:49.113529 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:49.113449    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce729ee-bcfb-48d3-86f7-02bf863a1ec2" path="/var/lib/kubelet/pods/cce729ee-bcfb-48d3-86f7-02bf863a1ec2/volumes"
Oct 30 14:51:49.160311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:49.160281    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:51:49.462257 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:49.462225    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75db85b96b-t7j2b"]
Oct 30 14:51:49.462726 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:51:49.462701    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe7f074_2d3f_44b8_9d3d_e54407143172.slice/crio-f01fbbe65ec886a15e519089ac40fc1b0cd50eaf12910745d9a23056fd20f301 WatchSource:0}: Error finding container f01fbbe65ec886a15e519089ac40fc1b0cd50eaf12910745d9a23056fd20f301: Status 404 returned error can't find the container with id f01fbbe65ec886a15e519089ac40fc1b0cd50eaf12910745d9a23056fd20f301
Oct 30 14:51:49.564138 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:49.564101    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75db85b96b-t7j2b" event={"ID":"abe7f074-2d3f-44b8-9d3d-e54407143172","Type":"ContainerStarted","Data":"f01fbbe65ec886a15e519089ac40fc1b0cd50eaf12910745d9a23056fd20f301"}
Oct 30 14:51:50.571963 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:50.571925    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75db85b96b-t7j2b" event={"ID":"abe7f074-2d3f-44b8-9d3d-e54407143172","Type":"ContainerStarted","Data":"c2060bc7bc49e4760ee743f169c5a781c9e2cc72b07e28fb4a652cfe32204c82"}
Oct 30 14:51:50.572400 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:50.572079    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:51:50.623520 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:51:50.623469    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75db85b96b-t7j2b" podStartSLOduration=2.236500637 podStartE2EDuration="2.623457185s" totalImagesPullingTime="386.956548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:50.620864642 +0000 UTC m=+3775.652763593" watchObservedRunningTime="2026-10-30 14:51:50.623457185 +0000 UTC m=+3775.655356156"
Oct 30 14:52:01.580722 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:01.580685    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75db85b96b-t7j2b"
Oct 30 14:52:03.320739 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:03.320701    4530 scope.go:122] "RemoveContainer" containerID="8dcf31c33893f296e1b6e54373a112d5f29bd814bba0e9483bd48389b462f4aa"
Oct 30 14:52:11.367047 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:52:11.367008    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:52:27.464053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:27.464014    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:52:41.365472 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:52:41.365392    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:52:41.706876 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.706803    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"]
Oct 30 14:52:41.710483 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.710462    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.728046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.728020    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"]
Oct 30 14:52:41.732051 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.732029    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.742471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.742448    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"]
Oct 30 14:52:41.762944 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.762920    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"]
Oct 30 14:52:41.782396 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782367    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.782535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782403    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/588b499d-571d-449a-9e75-126e4c84902c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.782535 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782436    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/588b499d-571d-449a-9e75-126e4c84902c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.782829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782570    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.782829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782605    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.782829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782626    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.782829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782714    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.782829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782749    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2c5de7-2259-4764-bca8-8811fe92c03d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.782829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782770    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.782829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782816    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.783262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782843    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.783262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782870    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/588b499d-571d-449a-9e75-126e4c84902c-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.783262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782915    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.783262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782951    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4q6h\" (UniqueName: \"kubernetes.io/projected/7d2c5de7-2259-4764-bca8-8811fe92c03d-kube-api-access-g4q6h\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.783262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782971    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.783262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.782988    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2c5de7-2259-4764-bca8-8811fe92c03d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.783262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.783006    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwhqx\" (UniqueName: \"kubernetes.io/projected/588b499d-571d-449a-9e75-126e4c84902c-kube-api-access-zwhqx\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.783262 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.783024    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2c5de7-2259-4764-bca8-8811fe92c03d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.883717 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.883687    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.883880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.883726    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/588b499d-571d-449a-9e75-126e4c84902c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.883880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.883747    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/588b499d-571d-449a-9e75-126e4c84902c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.883880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.883776    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.883880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.883796    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.883880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.883814    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.883880 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.883846    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.884249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884034    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2c5de7-2259-4764-bca8-8811fe92c03d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.884249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884087    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.884249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884117    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.884249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884141    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.884249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884164    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/588b499d-571d-449a-9e75-126e4c84902c-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.884249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884185    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.884249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884205    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.884249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884211    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.884249 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884218    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.884796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884252    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.884796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884466    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.884796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884494    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.884796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884523    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4q6h\" (UniqueName: \"kubernetes.io/projected/7d2c5de7-2259-4764-bca8-8811fe92c03d-kube-api-access-g4q6h\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.884796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884581    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.884796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884583    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/588b499d-571d-449a-9e75-126e4c84902c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.884796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884586    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.884796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884610    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2c5de7-2259-4764-bca8-8811fe92c03d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.884796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884653    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwhqx\" (UniqueName: \"kubernetes.io/projected/588b499d-571d-449a-9e75-126e4c84902c-kube-api-access-zwhqx\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.884796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884665    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.884796 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.884687    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2c5de7-2259-4764-bca8-8811fe92c03d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.887216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.885994    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2c5de7-2259-4764-bca8-8811fe92c03d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.887216 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.886765    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/588b499d-571d-449a-9e75-126e4c84902c-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.888062 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.888035    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/588b499d-571d-449a-9e75-126e4c84902c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.894768 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.894741    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2c5de7-2259-4764-bca8-8811fe92c03d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.898085 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.898063    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2c5de7-2259-4764-bca8-8811fe92c03d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.911150 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.911121    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/588b499d-571d-449a-9e75-126e4c84902c-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:41.911610 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.911588    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4q6h\" (UniqueName: \"kubernetes.io/projected/7d2c5de7-2259-4764-bca8-8811fe92c03d-kube-api-access-g4q6h\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.911701 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.911680    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2c5de7-2259-4764-bca8-8811fe92c03d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5gtn\" (UID: \"7d2c5de7-2259-4764-bca8-8811fe92c03d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:41.911829 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:41.911814    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwhqx\" (UniqueName: \"kubernetes.io/projected/588b499d-571d-449a-9e75-126e4c84902c-kube-api-access-zwhqx\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fb779\" (UID: \"588b499d-571d-449a-9e75-126e4c84902c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:42.024923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.024836    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:42.048457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.048432    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:42.345800 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.345753    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"]
Oct 30 14:52:42.358666 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:52:42.358639    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588b499d_571d_449a_9e75_126e4c84902c.slice/crio-2633fad389c41dc2e9425541d4ab6aaf08201ed0f5ef95c0a533d0f38f29776d WatchSource:0}: Error finding container 2633fad389c41dc2e9425541d4ab6aaf08201ed0f5ef95c0a533d0f38f29776d: Status 404 returned error can't find the container with id 2633fad389c41dc2e9425541d4ab6aaf08201ed0f5ef95c0a533d0f38f29776d
Oct 30 14:52:42.360864 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.360824    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:52:42.360977 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.360942    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:52:42.361046 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.360989    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:52:42.387778 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:52:42.387746    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2c5de7_2259_4764_bca8_8811fe92c03d.slice/crio-b4bfdd1a03d48e3a6ba1c4064bcb9b88072a3280a8c739e3fc194c22a8a4ebe9 WatchSource:0}: Error finding container b4bfdd1a03d48e3a6ba1c4064bcb9b88072a3280a8c739e3fc194c22a8a4ebe9: Status 404 returned error can't find the container with id b4bfdd1a03d48e3a6ba1c4064bcb9b88072a3280a8c739e3fc194c22a8a4ebe9
Oct 30 14:52:42.388410 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.388387    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"]
Oct 30 14:52:42.390117 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.390077    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:52:42.390197 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.390165    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:52:42.390237 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.390210    4530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 14:52:42.860192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.860149    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" event={"ID":"7d2c5de7-2259-4764-bca8-8811fe92c03d","Type":"ContainerStarted","Data":"0d524525094470bfae4d12dd75e0d8398b75027b8096b5ded7e818ecb5b7bfdb"}
Oct 30 14:52:42.860442 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.860217    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" event={"ID":"7d2c5de7-2259-4764-bca8-8811fe92c03d","Type":"ContainerStarted","Data":"b4bfdd1a03d48e3a6ba1c4064bcb9b88072a3280a8c739e3fc194c22a8a4ebe9"}
Oct 30 14:52:42.861819 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.861793    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" event={"ID":"588b499d-571d-449a-9e75-126e4c84902c","Type":"ContainerStarted","Data":"dc89037758cf7414f1d53bd9f6f6109d67f96c49a3c97294825fcfb26f7a7081"}
Oct 30 14:52:42.861940 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.861836    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" event={"ID":"588b499d-571d-449a-9e75-126e4c84902c","Type":"ContainerStarted","Data":"2633fad389c41dc2e9425541d4ab6aaf08201ed0f5ef95c0a533d0f38f29776d"}
Oct 30 14:52:42.966555 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.966500    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podStartSLOduration=1.966486564 podStartE2EDuration="1.966486564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:52:42.964438046 +0000 UTC m=+3827.996336986" watchObservedRunningTime="2026-10-30 14:52:42.966486564 +0000 UTC m=+3827.998385497"
Oct 30 14:52:42.966746 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:42.966697    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podStartSLOduration=1.966690742 podStartE2EDuration="1.966690742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:52:42.913874821 +0000 UTC m=+3827.945773832" watchObservedRunningTime="2026-10-30 14:52:42.966690742 +0000 UTC m=+3827.998589705"
Oct 30 14:52:43.027644 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:43.027601    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:43.028038 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:43.028013    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fb779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 14:52:43.028187 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:43.028165    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podUID="588b499d-571d-449a-9e75-126e4c84902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 14:52:43.049346 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:43.049315    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:43.050782 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:43.050757    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 14:52:43.050913 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:43.050808    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podUID="7d2c5de7-2259-4764-bca8-8811fe92c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 14:52:44.026149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:44.026109    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fb779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 14:52:44.026887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:44.026828    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podUID="588b499d-571d-449a-9e75-126e4c84902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 14:52:44.049368 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:44.049335    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 14:52:44.049525 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:44.049389    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podUID="7d2c5de7-2259-4764-bca8-8811fe92c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 14:52:45.015567 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:45.012794    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:52:45.030018 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:45.029961    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fb779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 14:52:45.030606 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:45.030024    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podUID="588b499d-571d-449a-9e75-126e4c84902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 14:52:45.048979 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:45.048949    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 14:52:45.049113 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:45.048995    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podUID="7d2c5de7-2259-4764-bca8-8811fe92c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 14:52:46.029149 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:46.029109    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fb779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 14:52:46.029509 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:46.029478    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podUID="588b499d-571d-449a-9e75-126e4c84902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 14:52:46.049457 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:46.049411    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 14:52:46.050012 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:46.049981    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podUID="7d2c5de7-2259-4764-bca8-8811fe92c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 14:52:47.026064 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:47.026019    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fb779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 14:52:47.026514 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:47.026480    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podUID="588b499d-571d-449a-9e75-126e4c84902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 14:52:47.049060 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:47.049032    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 14:52:47.049254 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:47.049235    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podUID="7d2c5de7-2259-4764-bca8-8811fe92c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 14:52:48.026059 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:48.026012    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fb779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 14:52:48.027189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:48.026712    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podUID="588b499d-571d-449a-9e75-126e4c84902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 14:52:48.049391 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:48.049363    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 14:52:48.049614 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:48.049587    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podUID="7d2c5de7-2259-4764-bca8-8811fe92c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 14:52:49.026043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:49.025941    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fb779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 30 14:52:49.026043 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:49.026027    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podUID="588b499d-571d-449a-9e75-126e4c84902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 30 14:52:49.049523 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:49.049471    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 14:52:49.049765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:49.049587    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podUID="7d2c5de7-2259-4764-bca8-8811fe92c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 14:52:51.080774 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:51.080680    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fb779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:52:51.081483 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:51.081424    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podUID="588b499d-571d-449a-9e75-126e4c84902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:52:51.091974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:51.091624    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:52:51.091974 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:51.091667    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podUID="7d2c5de7-2259-4764-bca8-8811fe92c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:52:51.478776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:51.478673    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-52g42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:52:51.478776 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:51.478754    4530 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-52g42" podUID="075cda4c-17d3-48d0-801c-0a0101cb3a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:52:51.853645 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:51.853609    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:52:52.083114 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:52.083069    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fb779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:52:52.083743 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:52.083690    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podUID="588b499d-571d-449a-9e75-126e4c84902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:52:52.092451 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:52.092420    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:52:52.092622 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:52.092465    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podUID="7d2c5de7-2259-4764-bca8-8811fe92c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:52:53.124764 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:53.124709    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fb779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:52:53.125452 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:53.125387    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podUID="588b499d-571d-449a-9e75-126e4c84902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:52:53.133741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:53.133696    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:52:53.133986 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:53.133938    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podUID="7d2c5de7-2259-4764-bca8-8811fe92c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:52:54.125794 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:54.125750    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fb779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:52:54.126299 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:54.125814    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podUID="588b499d-571d-449a-9e75-126e4c84902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:52:54.134513 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:54.134476    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:52:54.134647 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:54.134574    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podUID="7d2c5de7-2259-4764-bca8-8811fe92c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:52:55.167731 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:55.167681    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fb779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:52:55.168251 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:55.167750    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779" podUID="588b499d-571d-449a-9e75-126e4c84902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:52:55.175664 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:55.175638    4530 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 14:52:55.175816 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:55.175675    4530 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn" podUID="7d2c5de7-2259-4764-bca8-8811fe92c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 14:52:56.076167 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:56.076120    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:56.076565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:56.076528    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:56.077213 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:56.077189    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fb779"
Oct 30 14:52:56.142448 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:56.142415    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:56.142742 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:56.142714    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:52:56.143533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:52:56.143511    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5gtn"
Oct 30 14:53:07.889247 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:53:07.889203    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:53:11.378580 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:53:11.378275    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:53:23.567616 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:53:23.567585    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:53:41.366808 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:53:41.366770    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:53:55.876100 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:53:55.876057    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:53:56.193498 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:53:56.193416    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:53:56.198427 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:53:56.198404    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:53:56.198594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:53:56.198451    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:53:56.199429 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:53:56.199408    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:53:56.204048 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:53:56.204020    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:53:56.205036 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:53:56.205012    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:54:06.256697 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:54:06.256659    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:54:10.641369 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:54:10.641282    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:54:11.379079 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:54:11.379036    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:54:38.034403 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:54:38.034363    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:54:41.366221 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:54:41.366177    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:54:49.263716 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:54:49.263681    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:55:02.071797 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:02.071761    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"]
Oct 30 14:55:02.087628 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:02.087598    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532c6e25-x26fm"]
Oct 30 14:55:03.113859 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:03.113818    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c8a429-c99d-4a91-b4e5-e3f18274e20e" path="/var/lib/kubelet/pods/e9c8a429-c99d-4a91-b4e5-e3f18274e20e/volumes"
Oct 30 14:55:06.492891 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:06.492835    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:55:11.365884 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:55:11.365847    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:55:13.036772 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:13.036734    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"]
Oct 30 14:55:13.052657 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:13.052621    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-342ce888-mwhp8"]
Oct 30 14:55:13.113392 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:13.113360    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d586e5-be6a-4560-8e91-3fcf9fd13e04" path="/var/lib/kubelet/pods/94d586e5-be6a-4560-8e91-3fcf9fd13e04/volumes"
Oct 30 14:55:20.796202 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:20.796153    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:55:26.041580 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:26.041530    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"]
Oct 30 14:55:26.055565 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:26.055521    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-593c3d4d-5tvvg"]
Oct 30 14:55:27.033470 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:27.033436    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"]
Oct 30 14:55:27.045022 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:27.044990    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea14b59-qmhvl"]
Oct 30 14:55:27.113903 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:27.113874    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e610c9a-c5c7-426f-b543-e8c45cf56f8e" path="/var/lib/kubelet/pods/1e610c9a-c5c7-426f-b543-e8c45cf56f8e/volumes"
Oct 30 14:55:27.114328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:27.114313    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de876db-5913-45bb-988d-1a6617cd6575" path="/var/lib/kubelet/pods/4de876db-5913-45bb-988d-1a6617cd6575/volumes"
Oct 30 14:55:37.671055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:37.670973    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:55:41.366711 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:55:41.366669    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:55:53.735612 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:55:53.735565    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:56:03.375135 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:56:03.375098    4530 scope.go:122] "RemoveContainer" containerID="1683434a035bd3122fd53a64b781a2d22f554e43dad0a11a33c137a6c8fa65a3"
Oct 30 14:56:03.386322 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:56:03.386297    4530 scope.go:122] "RemoveContainer" containerID="d02ad38ffff6bfa49d31abf1cb586c4a435e97acf060a51096892ee600e216d0"
Oct 30 14:56:11.366017 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:56:11.365979    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:56:14.564781 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:56:14.564742    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:56:19.767854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:56:19.767819    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:56:36.677285 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:56:36.677246    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:56:41.367741 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:56:41.367690    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:56:41.648825 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:56:41.648731    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:57:03.412383 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:03.412345    4530 scope.go:122] "RemoveContainer" containerID="4839b1a8463fd122e3dbaab55164130e3c9cb7f17853460584a86028a6fcea06"
Oct 30 14:57:03.421449 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:03.421426    4530 scope.go:122] "RemoveContainer" containerID="16dc91ffeb88b0c2684ac115facb30fbe8bd2d60c8238fa744a5c9265cfe1e77"
Oct 30 14:57:03.429669 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:03.429650    4530 scope.go:122] "RemoveContainer" containerID="69d56bf086c3c7e188bc414b9d3847c8174b35ebde1bbeb44a62b92ae670e918"
Oct 30 14:57:03.437471 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:03.437452    4530 scope.go:122] "RemoveContainer" containerID="6bd8cb76760082b87af695c128f7c3ffa40b6af783f85adcd8e944ca7170b43c"
Oct 30 14:57:11.367130 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:57:11.367045    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:57:20.000123 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.000081    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"]
Oct 30 14:57:20.004132 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.004105    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.030349 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.030316    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"]
Oct 30 14:57:20.034813 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.034780    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.034984 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.034848    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.035056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.034977    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.035056 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.035039    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phg2f\" (UniqueName: \"kubernetes.io/projected/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-kube-api-access-phg2f\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.039846 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.039816    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-d3d611-serving-cert\""
Oct 30 14:57:20.039846 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.039824    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-54whp\""
Oct 30 14:57:20.040080 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.039982    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 14:57:20.136192 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.136144    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.136399 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:57:20.136296    4530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-serving-cert: secret "maas-api-e2e-worker-single-main-d3d611-serving-cert" not found
Oct 30 14:57:20.136399 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.136302    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.136399 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:57:20.136367    4530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-maas-api-tls podName:f1f86e3d-638a-4bfb-8e29-11f65e1f76f4 nodeName:}" failed. No retries permitted until 2026-10-30 14:57:20.636346688 +0000 UTC m=+4105.668245620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-maas-api-tls") pod "maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk" (UID: "f1f86e3d-638a-4bfb-8e29-11f65e1f76f4") : secret "maas-api-e2e-worker-single-main-d3d611-serving-cert" not found
Oct 30 14:57:20.136627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.136410    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.136627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.136472    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phg2f\" (UniqueName: \"kubernetes.io/projected/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-kube-api-access-phg2f\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.137029 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.137003    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.139025 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.139003    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.174244 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.174211    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phg2f\" (UniqueName: \"kubernetes.io/projected/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-kube-api-access-phg2f\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.644450 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.644408    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.647023 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.646990    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:20.918594 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:20.918477    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:21.224307 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:21.224275    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"]
Oct 30 14:57:21.224854 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:57:21.224828    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f86e3d_638a_4bfb_8e29_11f65e1f76f4.slice/crio-5647049133d053cb51082a505c86c0c90a76c108e1b79d54290e902dff5df478 WatchSource:0}: Error finding container 5647049133d053cb51082a505c86c0c90a76c108e1b79d54290e902dff5df478: Status 404 returned error can't find the container with id 5647049133d053cb51082a505c86c0c90a76c108e1b79d54290e902dff5df478
Oct 30 14:57:21.226634 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:21.226613    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 14:57:21.818603 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:21.818558    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk" event={"ID":"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4","Type":"ContainerStarted","Data":"fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a"}
Oct 30 14:57:21.818854 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:21.818827    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk" event={"ID":"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4","Type":"ContainerStarted","Data":"5647049133d053cb51082a505c86c0c90a76c108e1b79d54290e902dff5df478"}
Oct 30 14:57:21.819000 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:21.818863    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:21.870590 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:21.870441    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk" podStartSLOduration=2.500594329 podStartE2EDuration="2.870422901s" totalImagesPullingTime="369.828572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:57:21.866050409 +0000 UTC m=+4106.897949349" watchObservedRunningTime="2026-10-30 14:57:21.870422901 +0000 UTC m=+4106.902321840"
Oct 30 14:57:23.862881 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:23.862842    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:57:27.829311 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:27.829275    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:57:40.745815 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.745776    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"]
Oct 30 14:57:40.749357 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.749334    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.768361 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.768332    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"]
Oct 30 14:57:40.779682 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.779654    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d3d611\"/\"default-dockercfg-gffm2\""
Oct 30 14:57:40.779841 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.779679    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d3d611\"/\"simulator-single-main-d3d611-kserve-self-signed-certs\""
Oct 30 14:57:40.781119 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.781053    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d3d611\"/\"openshift-service-ca.crt\""
Oct 30 14:57:40.781266 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.781186    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d3d611\"/\"kube-root-ca.crt\""
Oct 30 14:57:40.838585 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.838529    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tmp-dir\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.838788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.838599    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tls-certs\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.838788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.838715    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-model-cache\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.838788 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.838781    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fs9f\" (UniqueName: \"kubernetes.io/projected/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-kube-api-access-8fs9f\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.838916 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.838900    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-dshm\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.838966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.838943    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-home\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.940031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.939993    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-model-cache\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.940031 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.940032    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fs9f\" (UniqueName: \"kubernetes.io/projected/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-kube-api-access-8fs9f\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.940312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.940095    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-dshm\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.940312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.940135    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-home\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.940312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.940185    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tmp-dir\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.940312 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.940221    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tls-certs\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.940674 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.940592    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-model-cache\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.940741 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.940688    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-home\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.940805 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.940748    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tmp-dir\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.942579 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.942520    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-dshm\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.942877 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.942858    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tls-certs\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:40.969270 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:40.969242    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fs9f\" (UniqueName: \"kubernetes.io/projected/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-kube-api-access-8fs9f\") pod \"simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") " pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:41.064462 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:41.063013    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:41.368814 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:57:41.368721    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:57:41.373402 ip-10-0-133-118 kubenswrapper[4530]: W1030 14:57:41.373370    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d38f3d4_66d8_434c_a05c_80b775ccd3fb.slice/crio-44d9a0cbc631e57b47c0f0d04b305c6c0f9a8d6c0e3e58b0af5b7ab835ca0af5 WatchSource:0}: Error finding container 44d9a0cbc631e57b47c0f0d04b305c6c0f9a8d6c0e3e58b0af5b7ab835ca0af5: Status 404 returned error can't find the container with id 44d9a0cbc631e57b47c0f0d04b305c6c0f9a8d6c0e3e58b0af5b7ab835ca0af5
Oct 30 14:57:41.373533 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:41.373408    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"]
Oct 30 14:57:41.948412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:41.948374    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr" event={"ID":"9d38f3d4-66d8-434c-a05c-80b775ccd3fb","Type":"ContainerStarted","Data":"9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d"}
Oct 30 14:57:41.948412 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:41.948425    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr" event={"ID":"9d38f3d4-66d8-434c-a05c-80b775ccd3fb","Type":"ContainerStarted","Data":"44d9a0cbc631e57b47c0f0d04b305c6c0f9a8d6c0e3e58b0af5b7ab835ca0af5"}
Oct 30 14:57:41.998189 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:41.998128    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr" podStartSLOduration=1.998113402 podStartE2EDuration="1.998113402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:57:41.995384313 +0000 UTC m=+4127.027283252" watchObservedRunningTime="2026-10-30 14:57:41.998113402 +0000 UTC m=+4127.030012341"
Oct 30 14:57:42.063195 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:42.063154    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:43.041830 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:43.041789    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:57:51.064053 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:51.064010    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:51.077201 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:51.077172    4530 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:57:52.014662 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:57:52.014627    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:58:08.159924 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:08.159884    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:58:11.366930 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:58:11.366891    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:58:25.749320 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:25.749282    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:58:30.344340 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:30.344293    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"]
Oct 30 14:58:36.773119 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:36.773083    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"]
Oct 30 14:58:36.773636 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:36.773353    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk" podUID="f1f86e3d-638a-4bfb-8e29-11f65e1f76f4" containerName="maas-api" containerID="cri-o://fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a" gracePeriod=30
Oct 30 14:58:37.258729 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.258701    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:58:37.286783 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.286750    4530 generic.go:356] "Generic (PLEG): container finished" podID="f1f86e3d-638a-4bfb-8e29-11f65e1f76f4" containerID="fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a" exitCode=0
Oct 30 14:58:37.286939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.286798    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"
Oct 30 14:58:37.286939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.286839    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk" event={"ID":"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4","Type":"ContainerDied","Data":"fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a"}
Oct 30 14:58:37.286939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.286895    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk" event={"ID":"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4","Type":"ContainerDied","Data":"5647049133d053cb51082a505c86c0c90a76c108e1b79d54290e902dff5df478"}
Oct 30 14:58:37.286939 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.286923    4530 scope.go:122] "RemoveContainer" containerID="fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a"
Oct 30 14:58:37.296437 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.296416    4530 scope.go:122] "RemoveContainer" containerID="fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a"
Oct 30 14:58:37.296702 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:58:37.296679    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a\": container with ID starting with fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a not found: ID does not exist" containerID="fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a"
Oct 30 14:58:37.296777 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.296711    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a"} err="failed to get container status \"fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a\": rpc error: code = NotFound desc = could not find container \"fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a\": container with ID starting with fe6b949f60cc2302a440acb43db7ffa151e912124a5df2468b6e477bf335279a not found: ID does not exist"
Oct 30 14:58:37.323194 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.323131    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-kube-api-access-phg2f\" (UniqueName: \"kubernetes.io/projected/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-kube-api-access-phg2f\") pod \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") "
Oct 30 14:58:37.323341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.323200    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-metrics-certs\") pod \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") "
Oct 30 14:58:37.323341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.323226    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-openshift-service-ca\") pod \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") "
Oct 30 14:58:37.323341 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.323268    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-maas-api-tls\") pod \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\" (UID: \"f1f86e3d-638a-4bfb-8e29-11f65e1f76f4\") "
Oct 30 14:58:37.323700 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.323664    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-openshift-service-ca" pod "f1f86e3d-638a-4bfb-8e29-11f65e1f76f4" (UID: "f1f86e3d-638a-4bfb-8e29-11f65e1f76f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:58:37.325765 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.325735    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-metrics-certs" pod "f1f86e3d-638a-4bfb-8e29-11f65e1f76f4" (UID: "f1f86e3d-638a-4bfb-8e29-11f65e1f76f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:58:37.325923 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.325901    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-maas-api-tls" pod "f1f86e3d-638a-4bfb-8e29-11f65e1f76f4" (UID: "f1f86e3d-638a-4bfb-8e29-11f65e1f76f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:58:37.328688 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.328661    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-kube-api-access-phg2f" pod "f1f86e3d-638a-4bfb-8e29-11f65e1f76f4" (UID: "f1f86e3d-638a-4bfb-8e29-11f65e1f76f4"). InnerVolumeSpecName "kube-api-access-phg2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:58:37.424887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.424845    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:58:37.424887 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.424884    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phg2f\" (UniqueName: \"kubernetes.io/projected/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-kube-api-access-phg2f\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:58:37.425093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.424898    4530 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-metrics-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:58:37.425093 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.424906    4530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:58:37.641125 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.641087    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"]
Oct 30 14:58:37.655142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:37.655113    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d3d611-754db494d4-s4hfk"]
Oct 30 14:58:39.061055 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:39.061012    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 14:58:39.113321 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:39.113277    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f86e3d-638a-4bfb-8e29-11f65e1f76f4" path="/var/lib/kubelet/pods/f1f86e3d-638a-4bfb-8e29-11f65e1f76f4/volumes"
Oct 30 14:58:41.366457 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:58:41.366416    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:58:45.364928 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.364859    4530 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr" podUID="9d38f3d4-66d8-434c-a05c-80b775ccd3fb" containerName="main" containerID="cri-o://9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d" gracePeriod=45
Oct 30 14:58:45.620863 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.620800    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:58:45.706999 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.706963    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tmp-dir\") pod \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") "
Oct 30 14:58:45.707218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.707023    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-model-cache\") pod \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") "
Oct 30 14:58:45.707218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.707099    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-home\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-home\") pod \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") "
Oct 30 14:58:45.707218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.707159    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-dshm\") pod \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") "
Oct 30 14:58:45.707218 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.707189    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tls-certs\") pod \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") "
Oct 30 14:58:45.707497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.707249    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-kube-api-access-8fs9f\" (UniqueName: \"kubernetes.io/projected/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-kube-api-access-8fs9f\") pod \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\" (UID: \"9d38f3d4-66d8-434c-a05c-80b775ccd3fb\") "
Oct 30 14:58:45.707497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.707261    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tmp-dir" pod "9d38f3d4-66d8-434c-a05c-80b775ccd3fb" (UID: "9d38f3d4-66d8-434c-a05c-80b775ccd3fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:58:45.707497 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.707331    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-model-cache" pod "9d38f3d4-66d8-434c-a05c-80b775ccd3fb" (UID: "9d38f3d4-66d8-434c-a05c-80b775ccd3fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:58:45.708004 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.707929    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-home" pod "9d38f3d4-66d8-434c-a05c-80b775ccd3fb" (UID: "9d38f3d4-66d8-434c-a05c-80b775ccd3fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:58:45.714245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.711702    4530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:58:45.714245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.711744    4530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:58:45.714245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.711759    4530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:58:45.714245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.712434    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-dshm" pod "9d38f3d4-66d8-434c-a05c-80b775ccd3fb" (UID: "9d38f3d4-66d8-434c-a05c-80b775ccd3fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:58:45.714245 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.712568    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tls-certs" pod "9d38f3d4-66d8-434c-a05c-80b775ccd3fb" (UID: "9d38f3d4-66d8-434c-a05c-80b775ccd3fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:58:45.719328 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.718026    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-kube-api-access-8fs9f" pod "9d38f3d4-66d8-434c-a05c-80b775ccd3fb" (UID: "9d38f3d4-66d8-434c-a05c-80b775ccd3fb"). InnerVolumeSpecName "kube-api-access-8fs9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:58:45.812808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.812777    4530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:58:45.812808 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.812805    4530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:58:45.813027 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:45.812822    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fs9f\" (UniqueName: \"kubernetes.io/projected/9d38f3d4-66d8-434c-a05c-80b775ccd3fb-kube-api-access-8fs9f\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 14:58:46.352885 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:46.352839    4530 generic.go:356] "Generic (PLEG): container finished" podID="9d38f3d4-66d8-434c-a05c-80b775ccd3fb" containerID="9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d" exitCode=0
Oct 30 14:58:46.353044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:46.352929    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"
Oct 30 14:58:46.353044 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:46.352927    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr" event={"ID":"9d38f3d4-66d8-434c-a05c-80b775ccd3fb","Type":"ContainerDied","Data":"9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d"}
Oct 30 14:58:46.353142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:46.353048    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr" event={"ID":"9d38f3d4-66d8-434c-a05c-80b775ccd3fb","Type":"ContainerDied","Data":"44d9a0cbc631e57b47c0f0d04b305c6c0f9a8d6c0e3e58b0af5b7ab835ca0af5"}
Oct 30 14:58:46.353142 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:46.353065    4530 scope.go:122] "RemoveContainer" containerID="9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d"
Oct 30 14:58:46.366793 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:46.366584    4530 scope.go:122] "RemoveContainer" containerID="9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d"
Oct 30 14:58:46.367059 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:58:46.366875    4530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d\": container with ID starting with 9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d not found: ID does not exist" containerID="9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d"
Oct 30 14:58:46.367059 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:46.366904    4530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d"} err="failed to get container status \"9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d\": rpc error: code = NotFound desc = could not find container \"9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d\": container with ID starting with 9f7f4c4c855a4dffcf5313764b962c032e1088dd8bea538be2670948307dde2d not found: ID does not exist"
Oct 30 14:58:46.413726 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:46.413697    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"]
Oct 30 14:58:46.427627 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:46.427600    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-d3d611/simulator-single-main-d3d611-kserve-6fcb9c667-2tzdr"]
Oct 30 14:58:47.113950 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:47.113915    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d38f3d4-66d8-434c-a05c-80b775ccd3fb" path="/var/lib/kubelet/pods/9d38f3d4-66d8-434c-a05c-80b775ccd3fb/volumes"
Oct 30 14:58:56.233966 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:56.233936    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:58:56.238862 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:56.238831    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:58:56.239653 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:56.239630    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:58:56.242316 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:56.242295    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 14:58:56.247082 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:56.247060    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 14:58:56.248047 ip-10-0-133-118 kubenswrapper[4530]: I1030 14:58:56.248029    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 14:59:11.367124 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:59:11.367084    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 14:59:41.365913 ip-10-0-133-118 kubenswrapper[4530]: E1030 14:59:41.365873    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:00:07.537579 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.537525    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"]
Oct 30 15:00:07.538055 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.537975    4530 cpu_manager.go:414] "containerMap: removing container" podUID="9d38f3d4-66d8-434c-a05c-80b775ccd3fb" containerName="main"
Oct 30 15:00:07.538055 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.537989    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d38f3d4-66d8-434c-a05c-80b775ccd3fb" containerName="main"
Oct 30 15:00:07.538055 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.538010    4530 cpu_manager.go:414] "containerMap: removing container" podUID="f1f86e3d-638a-4bfb-8e29-11f65e1f76f4" containerName="maas-api"
Oct 30 15:00:07.538055 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.538015    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f86e3d-638a-4bfb-8e29-11f65e1f76f4" containerName="maas-api"
Oct 30 15:00:07.538197 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.538100    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d38f3d4-66d8-434c-a05c-80b775ccd3fb" containerName="main"
Oct 30 15:00:07.538197 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.538109    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f86e3d-638a-4bfb-8e29-11f65e1f76f4" containerName="maas-api"
Oct 30 15:00:07.541164 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.541146    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"
Oct 30 15:00:07.550760 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.550731    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"]
Oct 30 15:00:07.560842 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.560820    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zdx8r\""
Oct 30 15:00:07.671019 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.670955    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk924\" (UniqueName: \"kubernetes.io/projected/678a50b5-6193-4fbb-9d78-0f733e8b8cca-kube-api-access-dk924\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4cn5g\" (UID: \"678a50b5-6193-4fbb-9d78-0f733e8b8cca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"
Oct 30 15:00:07.671019 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.671001    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/678a50b5-6193-4fbb-9d78-0f733e8b8cca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4cn5g\" (UID: \"678a50b5-6193-4fbb-9d78-0f733e8b8cca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"
Oct 30 15:00:07.771783 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.771750    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk924\" (UniqueName: \"kubernetes.io/projected/678a50b5-6193-4fbb-9d78-0f733e8b8cca-kube-api-access-dk924\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4cn5g\" (UID: \"678a50b5-6193-4fbb-9d78-0f733e8b8cca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"
Oct 30 15:00:07.771938 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.771798    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/678a50b5-6193-4fbb-9d78-0f733e8b8cca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4cn5g\" (UID: \"678a50b5-6193-4fbb-9d78-0f733e8b8cca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"
Oct 30 15:00:07.772175 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.772158    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/678a50b5-6193-4fbb-9d78-0f733e8b8cca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4cn5g\" (UID: \"678a50b5-6193-4fbb-9d78-0f733e8b8cca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"
Oct 30 15:00:07.799354 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.799330    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk924\" (UniqueName: \"kubernetes.io/projected/678a50b5-6193-4fbb-9d78-0f733e8b8cca-kube-api-access-dk924\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4cn5g\" (UID: \"678a50b5-6193-4fbb-9d78-0f733e8b8cca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"
Oct 30 15:00:07.855485 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:07.855461    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"
Oct 30 15:00:08.185010 ip-10-0-133-118 kubenswrapper[4530]: W1030 15:00:08.184983    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678a50b5_6193_4fbb_9d78_0f733e8b8cca.slice/crio-e74e174fa310c43da32e054b642bf95b395dd73f0eb0c63a05e3bb383f66d011 WatchSource:0}: Error finding container e74e174fa310c43da32e054b642bf95b395dd73f0eb0c63a05e3bb383f66d011: Status 404 returned error can't find the container with id e74e174fa310c43da32e054b642bf95b395dd73f0eb0c63a05e3bb383f66d011
Oct 30 15:00:08.185346 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:08.185322    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"]
Oct 30 15:00:08.790936 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:08.790892    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g" event={"ID":"678a50b5-6193-4fbb-9d78-0f733e8b8cca","Type":"ContainerStarted","Data":"025cac154db20ad2c71578c271c785fded67fe44e7a6c9f73601e12040fc79da"}
Oct 30 15:00:08.791387 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:08.790961    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g" event={"ID":"678a50b5-6193-4fbb-9d78-0f733e8b8cca","Type":"ContainerStarted","Data":"e74e174fa310c43da32e054b642bf95b395dd73f0eb0c63a05e3bb383f66d011"}
Oct 30 15:00:08.791387 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:08.791077    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"
Oct 30 15:00:08.879117 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:08.879065    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g" podStartSLOduration=1.879051514 podStartE2EDuration="1.879051514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:08.877626126 +0000 UTC m=+4273.909525065" watchObservedRunningTime="2026-10-30 15:00:08.879051514 +0000 UTC m=+4273.910950453"
Oct 30 15:00:11.365791 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:00:11.365746    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:00:19.797974 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:19.797942    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4cn5g"
Oct 30 15:00:26.181916 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:26.181883    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:00:41.366228 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:00:41.366182    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:00:57.553182 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:00:57.553144    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:01:11.365489 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:01:11.365453    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:01:32.784574 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:01:32.783932    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:01:41.367125 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:01:41.367042    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:01:46.967498 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:01:46.967456    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:02:01.789364 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:01.789320    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:02:11.368346 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:02:11.368305    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:02:13.792888 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.792843    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"]
Oct 30 15:02:13.796437 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.796415    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:13.818368 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.818336    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"]
Oct 30 15:02:13.827913 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.827867    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 15:02:13.828077 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.828035    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 15:02:13.852931 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.852895    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5hnh\" (UniqueName: \"kubernetes.io/projected/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-kube-api-access-m5hnh\") pod \"maas-api-revoke-sub-332f87b0-9xw7z\" (UID: \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:13.853058 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.853026    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-332f87b0-9xw7z\" (UID: \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:13.853102 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.853066    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-332f87b0-9xw7z\" (UID: \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:13.953923 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.953895    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-332f87b0-9xw7z\" (UID: \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:13.954073 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.953935    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-332f87b0-9xw7z\" (UID: \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:13.954073 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.953987    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5hnh\" (UniqueName: \"kubernetes.io/projected/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-kube-api-access-m5hnh\") pod \"maas-api-revoke-sub-332f87b0-9xw7z\" (UID: \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:13.954697 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.954672    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-332f87b0-9xw7z\" (UID: \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:13.956322 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.956303    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-332f87b0-9xw7z\" (UID: \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:13.981788 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:13.981755    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5hnh\" (UniqueName: \"kubernetes.io/projected/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-kube-api-access-m5hnh\") pod \"maas-api-revoke-sub-332f87b0-9xw7z\" (UID: \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:14.109713 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:14.109683    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:14.413862 ip-10-0-133-118 kubenswrapper[4530]: W1030 15:02:14.413829    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ef08b4_9d65_46fc_92d6_cb12bf8cd538.slice/crio-a6a36463a9c9eb27106dc0c2aba321809ff1cf4d2ca41a5b723e6a704ac4b234 WatchSource:0}: Error finding container a6a36463a9c9eb27106dc0c2aba321809ff1cf4d2ca41a5b723e6a704ac4b234: Status 404 returned error can't find the container with id a6a36463a9c9eb27106dc0c2aba321809ff1cf4d2ca41a5b723e6a704ac4b234
Oct 30 15:02:14.414058 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:14.414034    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"]
Oct 30 15:02:14.635492 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:14.635459    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z" event={"ID":"26ef08b4-9d65-46fc-92d6-cb12bf8cd538","Type":"ContainerStarted","Data":"a6a36463a9c9eb27106dc0c2aba321809ff1cf4d2ca41a5b723e6a704ac4b234"}
Oct 30 15:02:15.642666 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:15.642627    4530 generic.go:356] "Generic (PLEG): container finished" podID="26ef08b4-9d65-46fc-92d6-cb12bf8cd538" containerID="0ab2e155625fa24319c3d6c2ffddc7a5502456684dc461aacc10e3823765afc9" exitCode=0
Oct 30 15:02:15.642666 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:15.642667    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z" event={"ID":"26ef08b4-9d65-46fc-92d6-cb12bf8cd538","Type":"ContainerDied","Data":"0ab2e155625fa24319c3d6c2ffddc7a5502456684dc461aacc10e3823765afc9"}
Oct 30 15:02:16.783832 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:16.783806    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:16.882566 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:16.882514    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-openshift-ca\") pod \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\" (UID: \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\") "
Oct 30 15:02:16.882773 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:16.882611    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-xks-ca\") pod \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\" (UID: \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\") "
Oct 30 15:02:16.882859 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:16.882789    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-kube-api-access-m5hnh\" (UniqueName: \"kubernetes.io/projected/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-kube-api-access-m5hnh\") pod \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\" (UID: \"26ef08b4-9d65-46fc-92d6-cb12bf8cd538\") "
Oct 30 15:02:16.882921 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:16.882879    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-openshift-ca" pod "26ef08b4-9d65-46fc-92d6-cb12bf8cd538" (UID: "26ef08b4-9d65-46fc-92d6-cb12bf8cd538"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:02:16.883150 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:16.883129    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 15:02:16.884733 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:16.884707    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-xks-ca" pod "26ef08b4-9d65-46fc-92d6-cb12bf8cd538" (UID: "26ef08b4-9d65-46fc-92d6-cb12bf8cd538"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:02:16.884863 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:16.884829    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-kube-api-access-m5hnh" pod "26ef08b4-9d65-46fc-92d6-cb12bf8cd538" (UID: "26ef08b4-9d65-46fc-92d6-cb12bf8cd538"). InnerVolumeSpecName "kube-api-access-m5hnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:02:16.983772 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:16.983691    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 15:02:16.983772 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:16.983720    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5hnh\" (UniqueName: \"kubernetes.io/projected/26ef08b4-9d65-46fc-92d6-cb12bf8cd538-kube-api-access-m5hnh\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 15:02:17.654402 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:17.654363    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z" event={"ID":"26ef08b4-9d65-46fc-92d6-cb12bf8cd538","Type":"ContainerDied","Data":"a6a36463a9c9eb27106dc0c2aba321809ff1cf4d2ca41a5b723e6a704ac4b234"}
Oct 30 15:02:17.654612 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:17.654412    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"
Oct 30 15:02:17.654612 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:17.654423    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6a36463a9c9eb27106dc0c2aba321809ff1cf4d2ca41a5b723e6a704ac4b234"
Oct 30 15:02:24.829531 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:24.829498    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"]
Oct 30 15:02:24.830035 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:24.829941    4530 cpu_manager.go:414] "containerMap: removing container" podUID="26ef08b4-9d65-46fc-92d6-cb12bf8cd538" containerName="revoke-keys"
Oct 30 15:02:24.830035 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:24.829953    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ef08b4-9d65-46fc-92d6-cb12bf8cd538" containerName="revoke-keys"
Oct 30 15:02:24.830035 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:24.830026    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ef08b4-9d65-46fc-92d6-cb12bf8cd538" containerName="revoke-keys"
Oct 30 15:02:24.833405 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:24.833385    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:24.856427 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:24.856391    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"]
Oct 30 15:02:24.867354 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:24.867333    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 15:02:24.867466 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:24.867441    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\""
Oct 30 15:02:24.963890 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:24.963858    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w66d7\" (UniqueName: \"kubernetes.io/projected/348e1e03-ba95-421b-8f19-e5de7244e5a7-kube-api-access-w66d7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg\" (UID: \"348e1e03-ba95-421b-8f19-e5de7244e5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:24.964069 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:24.963897    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg\" (UID: \"348e1e03-ba95-421b-8f19-e5de7244e5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:24.964069 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:24.963920    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg\" (UID: \"348e1e03-ba95-421b-8f19-e5de7244e5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:25.064709 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:25.064673    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w66d7\" (UniqueName: \"kubernetes.io/projected/348e1e03-ba95-421b-8f19-e5de7244e5a7-kube-api-access-w66d7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg\" (UID: \"348e1e03-ba95-421b-8f19-e5de7244e5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:25.064882 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:25.064716    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg\" (UID: \"348e1e03-ba95-421b-8f19-e5de7244e5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:25.064882 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:25.064737    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg\" (UID: \"348e1e03-ba95-421b-8f19-e5de7244e5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:25.065408 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:25.065385    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg\" (UID: \"348e1e03-ba95-421b-8f19-e5de7244e5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:25.067175 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:25.067151    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg\" (UID: \"348e1e03-ba95-421b-8f19-e5de7244e5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:25.091485 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:25.091411    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w66d7\" (UniqueName: \"kubernetes.io/projected/348e1e03-ba95-421b-8f19-e5de7244e5a7-kube-api-access-w66d7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg\" (UID: \"348e1e03-ba95-421b-8f19-e5de7244e5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:25.147592 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:25.147558    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:25.283496 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:25.283458    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:02:25.478082 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:25.478050    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"]
Oct 30 15:02:25.478244 ip-10-0-133-118 kubenswrapper[4530]: W1030 15:02:25.478221    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348e1e03_ba95_421b_8f19_e5de7244e5a7.slice/crio-adbed968c46437fdbb24b0b681b6ef2a2fb899a385456b5141b7a9c7f470cf9c WatchSource:0}: Error finding container adbed968c46437fdbb24b0b681b6ef2a2fb899a385456b5141b7a9c7f470cf9c: Status 404 returned error can't find the container with id adbed968c46437fdbb24b0b681b6ef2a2fb899a385456b5141b7a9c7f470cf9c
Oct 30 15:02:25.698898 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:25.698865    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg" event={"ID":"348e1e03-ba95-421b-8f19-e5de7244e5a7","Type":"ContainerStarted","Data":"adbed968c46437fdbb24b0b681b6ef2a2fb899a385456b5141b7a9c7f470cf9c"}
Oct 30 15:02:26.706254 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:26.706173    4530 generic.go:356] "Generic (PLEG): container finished" podID="348e1e03-ba95-421b-8f19-e5de7244e5a7" containerID="b6f787cf56244119e8f866171897abecdce1ff63b7a9fab5aa50edda6ab6be92" exitCode=0
Oct 30 15:02:26.706809 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:26.706301    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg" event={"ID":"348e1e03-ba95-421b-8f19-e5de7244e5a7","Type":"ContainerDied","Data":"b6f787cf56244119e8f866171897abecdce1ff63b7a9fab5aa50edda6ab6be92"}
Oct 30 15:02:27.875375 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:27.875346    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:27.993908 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:27.993821    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348e1e03-ba95-421b-8f19-e5de7244e5a7-kube-api-access-w66d7\" (UniqueName: \"kubernetes.io/projected/348e1e03-ba95-421b-8f19-e5de7244e5a7-kube-api-access-w66d7\") pod \"348e1e03-ba95-421b-8f19-e5de7244e5a7\" (UID: \"348e1e03-ba95-421b-8f19-e5de7244e5a7\") "
Oct 30 15:02:27.994083 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:27.993949    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-openshift-ca\") pod \"348e1e03-ba95-421b-8f19-e5de7244e5a7\" (UID: \"348e1e03-ba95-421b-8f19-e5de7244e5a7\") "
Oct 30 15:02:27.994083 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:27.993979    4530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-xks-ca\") pod \"348e1e03-ba95-421b-8f19-e5de7244e5a7\" (UID: \"348e1e03-ba95-421b-8f19-e5de7244e5a7\") "
Oct 30 15:02:27.994298 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:27.994261    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-openshift-ca" pod "348e1e03-ba95-421b-8f19-e5de7244e5a7" (UID: "348e1e03-ba95-421b-8f19-e5de7244e5a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 15:02:27.996054 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:27.996022    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-xks-ca" pod "348e1e03-ba95-421b-8f19-e5de7244e5a7" (UID: "348e1e03-ba95-421b-8f19-e5de7244e5a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 15:02:27.996161 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:27.996077    4530 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348e1e03-ba95-421b-8f19-e5de7244e5a7-kube-api-access-w66d7" pod "348e1e03-ba95-421b-8f19-e5de7244e5a7" (UID: "348e1e03-ba95-421b-8f19-e5de7244e5a7"). InnerVolumeSpecName "kube-api-access-w66d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 15:02:28.095091 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:28.095063    4530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w66d7\" (UniqueName: \"kubernetes.io/projected/348e1e03-ba95-421b-8f19-e5de7244e5a7-kube-api-access-w66d7\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 15:02:28.095091 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:28.095089    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-openshift-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 15:02:28.095259 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:28.095099    4530 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348e1e03-ba95-421b-8f19-e5de7244e5a7-maas-api-xks-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\""
Oct 30 15:02:28.719561 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:28.719507    4530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"
Oct 30 15:02:28.719561 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:28.719534    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg" event={"ID":"348e1e03-ba95-421b-8f19-e5de7244e5a7","Type":"ContainerDied","Data":"adbed968c46437fdbb24b0b681b6ef2a2fb899a385456b5141b7a9c7f470cf9c"}
Oct 30 15:02:28.719777 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:28.719597    4530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adbed968c46437fdbb24b0b681b6ef2a2fb899a385456b5141b7a9c7f470cf9c"
Oct 30 15:02:39.072047 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:39.072002    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"]
Oct 30 15:02:39.075636 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:39.075599    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ef0c7-pvbfg"]
Oct 30 15:02:39.114633 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:39.114599    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348e1e03-ba95-421b-8f19-e5de7244e5a7" path="/var/lib/kubelet/pods/348e1e03-ba95-421b-8f19-e5de7244e5a7/volumes"
Oct 30 15:02:40.743747 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:40.743700    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:02:41.367143 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:02:41.367099    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:02:53.172365 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:02:53.172323    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:03:11.366452 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:03:11.366350    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:03:20.682015 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:03:20.681974    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:03:41.369105 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:03:41.369065    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:03:46.342154 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:03:46.342115    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:03:56.275908 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:03:56.275878    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 15:03:56.282428 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:03:56.282400    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 15:03:56.284218 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:03:56.284188    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 15:03:56.286510 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:03:56.286489    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 15:03:56.291273 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:03:56.291251    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 15:03:56.292200 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:03:56.292181    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 15:03:58.449027 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:03:58.448994    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:04:06.441579 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:06.441527    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:04:11.365435 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:04:11.365394    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:04:16.845410 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:16.845364    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:04:21.354502 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:21.354466    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:04:39.375012 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:39.374930    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:04:41.367190 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:04:41.367147    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:04:47.855727 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.855693    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"]
Oct 30 15:04:47.856339 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.856160    4530 cpu_manager.go:414] "containerMap: removing container" podUID="348e1e03-ba95-421b-8f19-e5de7244e5a7" containerName="revoke-keys"
Oct 30 15:04:47.856339 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.856175    4530 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348e1e03-ba95-421b-8f19-e5de7244e5a7" containerName="revoke-keys"
Oct 30 15:04:47.856339 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.856250    4530 memory_manager.go:365] "RemoveStaleState removing state" podUID="348e1e03-ba95-421b-8f19-e5de7244e5a7" containerName="revoke-keys"
Oct 30 15:04:47.859618 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.859599    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:47.885127 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.885097    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"]
Oct 30 15:04:47.892429 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.892402    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 15:04:47.892584 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.892455    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-54whp\""
Oct 30 15:04:47.892584 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.892486    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 15:04:47.928040 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.928001    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c82ee5-78d6-42fc-8f97-367d6c5a8257-maas-api-tls\") pod \"maas-api-595cc648c-7wzvd\" (UID: \"b7c82ee5-78d6-42fc-8f97-367d6c5a8257\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:47.928214 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.928131    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbt7\" (UniqueName: \"kubernetes.io/projected/b7c82ee5-78d6-42fc-8f97-367d6c5a8257-kube-api-access-lxbt7\") pod \"maas-api-595cc648c-7wzvd\" (UID: \"b7c82ee5-78d6-42fc-8f97-367d6c5a8257\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:47.928292 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.928227    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c82ee5-78d6-42fc-8f97-367d6c5a8257-openshift-service-ca\") pod \"maas-api-595cc648c-7wzvd\" (UID: \"b7c82ee5-78d6-42fc-8f97-367d6c5a8257\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:47.928362 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:47.928289    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c82ee5-78d6-42fc-8f97-367d6c5a8257-metrics-certs\") pod \"maas-api-595cc648c-7wzvd\" (UID: \"b7c82ee5-78d6-42fc-8f97-367d6c5a8257\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:48.029084 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:48.029054    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c82ee5-78d6-42fc-8f97-367d6c5a8257-openshift-service-ca\") pod \"maas-api-595cc648c-7wzvd\" (UID: \"b7c82ee5-78d6-42fc-8f97-367d6c5a8257\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:48.029084 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:48.029087    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c82ee5-78d6-42fc-8f97-367d6c5a8257-metrics-certs\") pod \"maas-api-595cc648c-7wzvd\" (UID: \"b7c82ee5-78d6-42fc-8f97-367d6c5a8257\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:48.029263 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:48.029104    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c82ee5-78d6-42fc-8f97-367d6c5a8257-maas-api-tls\") pod \"maas-api-595cc648c-7wzvd\" (UID: \"b7c82ee5-78d6-42fc-8f97-367d6c5a8257\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:48.029263 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:48.029191    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbt7\" (UniqueName: \"kubernetes.io/projected/b7c82ee5-78d6-42fc-8f97-367d6c5a8257-kube-api-access-lxbt7\") pod \"maas-api-595cc648c-7wzvd\" (UID: \"b7c82ee5-78d6-42fc-8f97-367d6c5a8257\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:48.029839 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:48.029817    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c82ee5-78d6-42fc-8f97-367d6c5a8257-openshift-service-ca\") pod \"maas-api-595cc648c-7wzvd\" (UID: \"b7c82ee5-78d6-42fc-8f97-367d6c5a8257\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:48.031812 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:48.031788    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c82ee5-78d6-42fc-8f97-367d6c5a8257-metrics-certs\") pod \"maas-api-595cc648c-7wzvd\" (UID: \"b7c82ee5-78d6-42fc-8f97-367d6c5a8257\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:48.031916 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:48.031789    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c82ee5-78d6-42fc-8f97-367d6c5a8257-maas-api-tls\") pod \"maas-api-595cc648c-7wzvd\" (UID: \"b7c82ee5-78d6-42fc-8f97-367d6c5a8257\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:48.056159 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:48.056131    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbt7\" (UniqueName: \"kubernetes.io/projected/b7c82ee5-78d6-42fc-8f97-367d6c5a8257-kube-api-access-lxbt7\") pod \"maas-api-595cc648c-7wzvd\" (UID: \"b7c82ee5-78d6-42fc-8f97-367d6c5a8257\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:48.172568 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:48.172448    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:48.476232 ip-10-0-133-118 kubenswrapper[4530]: W1030 15:04:48.476202    4530 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c82ee5_78d6_42fc_8f97_367d6c5a8257.slice/crio-17d2d446711ab54cf75c586242aa9e19295b69f85a603dbb51cbeccd50d1ffa2 WatchSource:0}: Error finding container 17d2d446711ab54cf75c586242aa9e19295b69f85a603dbb51cbeccd50d1ffa2: Status 404 returned error can't find the container with id 17d2d446711ab54cf75c586242aa9e19295b69f85a603dbb51cbeccd50d1ffa2
Oct 30 15:04:48.477695 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:48.477673    4530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 15:04:48.478854 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:48.478831    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"]
Oct 30 15:04:48.793084 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:48.792998    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd" event={"ID":"b7c82ee5-78d6-42fc-8f97-367d6c5a8257","Type":"ContainerStarted","Data":"17d2d446711ab54cf75c586242aa9e19295b69f85a603dbb51cbeccd50d1ffa2"}
Oct 30 15:04:49.801506 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:49.801459    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd" event={"ID":"b7c82ee5-78d6-42fc-8f97-367d6c5a8257","Type":"ContainerStarted","Data":"882e2620566f5455e06180c07df236bc55f9751adf19ae31a0081d22c8e436ac"}
Oct 30 15:04:49.801962 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:49.801653    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:04:49.851286 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:49.851239    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd" podStartSLOduration=2.519488683 podStartE2EDuration="2.851223896s" totalImagesPullingTime="331.735213ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:49.848989625 +0000 UTC m=+4554.880888564" watchObservedRunningTime="2026-10-30 15:04:49.851223896 +0000 UTC m=+4554.883122835"
Oct 30 15:04:55.812597 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:04:55.812536    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-7wzvd"
Oct 30 15:05:11.366011 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:05:11.365973    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:05:41.366879 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:05:41.366837    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:05:50.476402 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:05:50.476364    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:06:11.366459 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:06:11.366370    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:06:17.866387 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:06:17.866348    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:06:23.222565 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:06:23.222300    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:06:40.355419 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:06:40.355372    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:06:41.367346 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:06:41.367300    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:07:01.942484 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:07:01.942444    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:07:11.366617 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:07:11.366572    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:07:17.034746 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:07:17.034707    4530 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"]
Oct 30 15:07:17.049270 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:07:17.049234    4530 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332f87b0-9xw7z"]
Oct 30 15:07:17.114124 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:07:17.114087    4530 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ef08b4-9d65-46fc-92d6-cb12bf8cd538" path="/var/lib/kubelet/pods/26ef08b4-9d65-46fc-92d6-cb12bf8cd538/volumes"
Oct 30 15:07:19.247264 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:07:19.247223    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lr8xc"]
Oct 30 15:07:41.367592 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:07:41.367499    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:07:50.039762 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:07:50.039731    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5bd6859554-ttz9m_0490f682-51d0-4e20-a9cb-9cb1940ebb76/manager/0.log"
Oct 30 15:07:50.965230 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:07:50.965200    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75db85b96b-t7j2b_abe7f074-2d3f-44b8-9d3d-e54407143172/manager/0.log"
Oct 30 15:07:51.613036 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:07:51.613006    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-k44b2_21ca8cc1-6a8a-4880-849b-a5534fc4ead1/manager/0.log"
Oct 30 15:07:51.836963 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:07:51.836930    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d55fbd5c9-2nm4s_2588566f-38bd-4f85-9190-2b8cce575c23/copy-manifests/0.log"
Oct 30 15:07:51.870115 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:07:51.870048    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d55fbd5c9-2nm4s_2588566f-38bd-4f85-9190-2b8cce575c23/manager/0.log"
Oct 30 15:08:03.660740 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:03.660708    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-7wzvd_b7c82ee5-78d6-42fc-8f97-367d6c5a8257/maas-api/0.log"
Oct 30 15:08:04.962792 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:04.962760    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-62qgf_f937b72b-9f30-4465-bb35-f17afd3c5335/postgres/0.log"
Oct 30 15:08:06.759821 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:06.759785    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-rpn8z_4b882a05-e44a-493d-86e6-171a423c8e2f/manager/0.log"
Oct 30 15:08:07.395474 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:07.395440    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-4cn5g_678a50b5-6193-4fbb-9d78-0f733e8b8cca/manager/0.log"
Oct 30 15:08:07.621756 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:07.621722    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lr8xc_3ddceb78-7035-4583-a501-3552ad3bc858/limitador/0.log"
Oct 30 15:08:08.792825 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:08.792793    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hdsbg_2e761f65-785b-45e0-90df-5ea5697d281b/discovery/0.log"
Oct 30 15:08:09.213794 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:09.213746    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8996685d-q52jz_cba183dd-ad5b-469f-9058-67196f9557e5/kube-auth-proxy/0.log"
Oct 30 15:08:09.406792 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:09.406766    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-52g42_075cda4c-17d3-48d0-801c-0a0101cb3a5b/istio-proxy/0.log"
Oct 30 15:08:09.968974 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:09.968938    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-d9pw9_b9c516e0-9b3d-4398-a08b-c9db1a762db1/istio-proxy/0.log"
Oct 30 15:08:10.364488 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:10.364459    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-fb779_588b499d-571d-449a-9e75-126e4c84902c/istio-proxy/0.log"
Oct 30 15:08:11.146874 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:11.146841    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-p5gtn_7d2c5de7-2259-4764-bca8-8811fe92c03d/istio-proxy/0.log"
Oct 30 15:08:11.345493 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:11.345463    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-g99dv_08de2a9c-9802-4b72-a844-58f650b6e573/payload-pre-processing/0.log"
Oct 30 15:08:11.365868 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:08:11.365836    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:08:11.527695 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:11.527625    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9dw4d_c340cff3-c8aa-40df-869a-5d5bf4c7210b/payload-processing/0.log"
Oct 30 15:08:22.556095 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:22.556065    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx_6abb2398-ea14-431b-be41-1008dfd3fbf2/main/0.log"
Oct 30 15:08:22.969724 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:22.969645    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-zvnwp_ccdbeeb1-2575-498d-9119-89c6714b3f51/main/0.log"
Oct 30 15:08:23.384839 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:23.384809    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v_b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0/main/0.log"
Oct 30 15:08:41.365794 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:08:41.365759    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:08:44.599460 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:44.599431    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5bd6859554-ttz9m_0490f682-51d0-4e20-a9cb-9cb1940ebb76/manager/0.log"
Oct 30 15:08:45.501378 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:45.501349    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75db85b96b-t7j2b_abe7f074-2d3f-44b8-9d3d-e54407143172/manager/0.log"
Oct 30 15:08:46.162358 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:46.162327    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-k44b2_21ca8cc1-6a8a-4880-849b-a5534fc4ead1/manager/0.log"
Oct 30 15:08:46.424672 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:46.424597    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d55fbd5c9-2nm4s_2588566f-38bd-4f85-9190-2b8cce575c23/copy-manifests/0.log"
Oct 30 15:08:46.459029 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:46.459005    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d55fbd5c9-2nm4s_2588566f-38bd-4f85-9190-2b8cce575c23/manager/0.log"
Oct 30 15:08:47.936061 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:47.936034    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-7wzvd_b7c82ee5-78d6-42fc-8f97-367d6c5a8257/maas-api/0.log"
Oct 30 15:08:49.377975 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:49.377948    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-62qgf_f937b72b-9f30-4465-bb35-f17afd3c5335/postgres/0.log"
Oct 30 15:08:50.798690 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:50.798662    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-rpn8z_4b882a05-e44a-493d-86e6-171a423c8e2f/manager/0.log"
Oct 30 15:08:56.322445 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:56.322409    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 15:08:56.327623 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:56.327596    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 15:08:56.328628 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:56.328604    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 15:08:56.333180 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:56.333156    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c699bbw_332a7113-4b99-4e44-8749-d9201567c64f/extract/0.log"
Oct 30 15:08:56.338652 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:56.338624    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tvrl_bab1137f-277b-48ec-81b0-60533ca6d832/extract/0.log"
Oct 30 15:08:56.347500 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:08:56.347464    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hb8zg_b9d56842-22b5-45ed-b1e9-5d1bc4686454/extract/0.log"
Oct 30 15:09:01.432201 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:01.432172    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-4cn5g_678a50b5-6193-4fbb-9d78-0f733e8b8cca/manager/0.log"
Oct 30 15:09:01.717648 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:01.717575    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lr8xc_3ddceb78-7035-4583-a501-3552ad3bc858/limitador/0.log"
Oct 30 15:09:03.649331 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:03.649300    4530 scope.go:122] "RemoveContainer" containerID="0ab2e155625fa24319c3d6c2ffddc7a5502456684dc461aacc10e3823765afc9"
Oct 30 15:09:03.658088 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:03.658067    4530 scope.go:122] "RemoveContainer" containerID="b6f787cf56244119e8f866171897abecdce1ff63b7a9fab5aa50edda6ab6be92"
Oct 30 15:09:11.365956 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:09:11.365873    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:09:12.713906 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:12.713876    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hdsbg_2e761f65-785b-45e0-90df-5ea5697d281b/discovery/0.log"
Oct 30 15:09:13.161398 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:13.161370    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8996685d-q52jz_cba183dd-ad5b-469f-9058-67196f9557e5/kube-auth-proxy/0.log"
Oct 30 15:09:13.362184 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:13.362154    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-52g42_075cda4c-17d3-48d0-801c-0a0101cb3a5b/istio-proxy/0.log"
Oct 30 15:09:13.984989 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:13.984955    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-d9pw9_b9c516e0-9b3d-4398-a08b-c9db1a762db1/istio-proxy/0.log"
Oct 30 15:09:14.401888 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:14.401862    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-fb779_588b499d-571d-449a-9e75-126e4c84902c/istio-proxy/0.log"
Oct 30 15:09:15.228517 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:15.228487    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-p5gtn_7d2c5de7-2259-4764-bca8-8811fe92c03d/istio-proxy/0.log"
Oct 30 15:09:15.429868 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:15.429838    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-g99dv_08de2a9c-9802-4b72-a844-58f650b6e573/payload-pre-processing/0.log"
Oct 30 15:09:15.637334 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:15.637304    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9dw4d_c340cff3-c8aa-40df-869a-5d5bf4c7210b/payload-processing/0.log"
Oct 30 15:09:16.398854 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:16.398827    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-h2vwx_6abb2398-ea14-431b-be41-1008dfd3fbf2/main/0.log"
Oct 30 15:09:26.814163 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:26.814132    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-zvnwp_ccdbeeb1-2575-498d-9119-89c6714b3f51/main/0.log"
Oct 30 15:09:27.220333 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:09:27.220262    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xr57v_b5ab7ea0-dce1-44df-b4f6-af8e1564c7c0/main/0.log"
Oct 30 15:09:41.365846 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:09:41.365805    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:10:11.366307 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:10:11.366271    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:10:41.365985 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:10:41.365908    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:10:42.138373 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:10:42.138343    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75db85b96b-t7j2b_abe7f074-2d3f-44b8-9d3d-e54407143172/manager/0.log"
Oct 30 15:10:42.459886 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:10:42.459810    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75db85b96b-t7j2b_abe7f074-2d3f-44b8-9d3d-e54407143172/manager/0.log"
Oct 30 15:10:43.202461 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:10:43.202434    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9dw4d_c340cff3-c8aa-40df-869a-5d5bf4c7210b/payload-processing/0.log"
Oct 30 15:10:43.393995 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:10:43.393969    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9dw4d_c340cff3-c8aa-40df-869a-5d5bf4c7210b/payload-processing/0.log"
Oct 30 15:11:11.366124 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:11:11.366082    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:11:27.049763 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:27.049728    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75db85b96b-t7j2b_abe7f074-2d3f-44b8-9d3d-e54407143172/manager/0.log"
Oct 30 15:11:27.348246 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:27.348172    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75db85b96b-t7j2b_abe7f074-2d3f-44b8-9d3d-e54407143172/manager/0.log"
Oct 30 15:11:28.086766 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:28.086737    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9dw4d_c340cff3-c8aa-40df-869a-5d5bf4c7210b/payload-processing/0.log"
Oct 30 15:11:28.273742 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:28.273713    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9dw4d_c340cff3-c8aa-40df-869a-5d5bf4c7210b/payload-processing/0.log"
Oct 30 15:11:33.032098 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:33.032068    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2w5dd_f0bbec59-52f9-4dc6-b24b-169f666c08ae/global-pull-secret-syncer/0.log"
Oct 30 15:11:34.235297 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:34.235265    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sctxm_efb9f1c7-f4ad-4ab4-a6a0-30a6509f96f2/konnectivity-agent/0.log"
Oct 30 15:11:34.632118 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:34.632086    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-118.ec2.internal_e329b8d309e1f8141c44f91d4bb26de8/haproxy/0.log"
Oct 30 15:11:40.840549 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:40.840499    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-rpn8z_4b882a05-e44a-493d-86e6-171a423c8e2f/manager/0.log"
Oct 30 15:11:41.251597 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:41.251519    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-4cn5g_678a50b5-6193-4fbb-9d78-0f733e8b8cca/manager/0.log"
Oct 30 15:11:41.367534 ip-10-0-133-118 kubenswrapper[4530]: E1030 15:11:41.367500    4530 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pb77c" containerName="extractor"
Oct 30 15:11:41.368838 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:41.368811    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lr8xc_3ddceb78-7035-4583-a501-3552ad3bc858/limitador/0.log"
Oct 30 15:11:42.192241 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.192210    4530 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"]
Oct 30 15:11:42.195923 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.195898    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.222596 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.222569    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"]
Oct 30 15:11:42.232048 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.232026    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-48dpl\"/\"kube-root-ca.crt\""
Oct 30 15:11:42.232189 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.232047    4530 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-48dpl\"/\"openshift-service-ca.crt\""
Oct 30 15:11:42.254341 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.254313    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-proc\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.254449 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.254350    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmqf\" (UniqueName: \"kubernetes.io/projected/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-kube-api-access-dmmqf\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.254493 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.254469    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-podres\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.254536 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.254500    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-sys\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.254536 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.254524    4530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-lib-modules\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.299724 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.299697    4530 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-48dpl\"/\"default-dockercfg-dvdxx\""
Oct 30 15:11:42.355831 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.355801    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-lib-modules\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.355953 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.355857    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-proc\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.355953 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.355880    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmqf\" (UniqueName: \"kubernetes.io/projected/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-kube-api-access-dmmqf\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.355953 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.355931    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-podres\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.355953 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.355937    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-lib-modules\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.356127 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.355984    4530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-sys\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.356127 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.355998    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-podres\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.356127 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.356002    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-proc\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.356127 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.356055    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-sys\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.382139 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.382106    4530 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmqf\" (UniqueName: \"kubernetes.io/projected/4c700c91-1bb9-4b3e-816e-ec8b85f4aee9-kube-api-access-dmmqf\") pod \"perf-node-gather-daemonset-rncv9\" (UID: \"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.508782 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.508697    4530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:42.833558 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:42.833502    4530 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"]
Oct 30 15:11:43.198295 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:43.198209    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9" event={"ID":"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9","Type":"ContainerStarted","Data":"ea140d933f38becbf8b2b811c51bc1b743da825ef61bb12030175302a23bea07"}
Oct 30 15:11:43.198295 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:43.198263    4530 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9" event={"ID":"4c700c91-1bb9-4b3e-816e-ec8b85f4aee9","Type":"ContainerStarted","Data":"b3bcbd4bc12dfc628a4d26bee2f9753f42d2728796cfa02b177f3330686a9d54"}
Oct 30 15:11:43.198738 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:43.198310    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:44.488093 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:44.488054    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-8kr8n_fe7de227-b792-4f2d-9328-b73420af695c/monitoring-plugin/0.log"
Oct 30 15:11:44.620299 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:44.620274    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-28t2s_2f4f2337-c31d-44cb-b97f-707b5d5d335f/node-exporter/0.log"
Oct 30 15:11:44.757036 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:44.756971    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-28t2s_2f4f2337-c31d-44cb-b97f-707b5d5d335f/kube-rbac-proxy/0.log"
Oct 30 15:11:44.887046 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:44.887020    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-28t2s_2f4f2337-c31d-44cb-b97f-707b5d5d335f/init-textfile/0.log"
Oct 30 15:11:45.751933 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:45.751900    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wrgg2_97ed0e87-c8d4-4fab-aa32-8c1da29fce5b/kube-rbac-proxy-main/0.log"
Oct 30 15:11:45.876224 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:45.876196    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wrgg2_97ed0e87-c8d4-4fab-aa32-8c1da29fce5b/kube-rbac-proxy-self/0.log"
Oct 30 15:11:46.006083 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:46.006013    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wrgg2_97ed0e87-c8d4-4fab-aa32-8c1da29fce5b/openshift-state-metrics/0.log"
Oct 30 15:11:47.402451 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:47.402423    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-zq9x4_5c1b1ee0-e5fc-4ce7-b3ad-d9880a5db41f/prometheus-operator-admission-webhook/0.log"
Oct 30 15:11:47.556614 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:47.556573    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bb946c6c-pqnnx_44b72bcd-b4bc-4a51-9978-d921191ef537/telemeter-client/0.log"
Oct 30 15:11:47.681935 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:47.681867    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bb946c6c-pqnnx_44b72bcd-b4bc-4a51-9978-d921191ef537/reload/0.log"
Oct 30 15:11:47.809478 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:47.809449    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bb946c6c-pqnnx_44b72bcd-b4bc-4a51-9978-d921191ef537/kube-rbac-proxy/0.log"
Oct 30 15:11:49.213475 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:49.213444    4530 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9"
Oct 30 15:11:49.261851 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:49.261799    4530 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-rncv9" podStartSLOduration=7.261784976 podStartE2EDuration="7.261784976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:43.244218705 +0000 UTC m=+4968.276117644" watchObservedRunningTime="2026-10-30 15:11:49.261784976 +0000 UTC m=+4974.293683915"
Oct 30 15:11:52.009324 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:52.009295    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-79d7dd84cd-p798f_c6e4eb76-fb65-4540-941f-1b418427c124/console/0.log"
Oct 30 15:11:52.156154 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:52.156122    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-cls76_12678c60-0b95-404d-aa2d-a10c0ccf91f4/download-server/0.log"
Oct 30 15:11:56.198342 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:56.198312    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nr98w_b4819261-bdd9-419e-abd3-15fa171de861/dns/0.log"
Oct 30 15:11:56.330767 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:56.330737    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nr98w_b4819261-bdd9-419e-abd3-15fa171de861/kube-rbac-proxy/0.log"
Oct 30 15:11:56.581449 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:56.581412    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjg7z_62977d18-4b0e-4e1e-ba64-bcda9ca2095e/dns-node-resolver/0.log"
Oct 30 15:11:57.780136 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:57.780097    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5fb64c5687-zhdxl_9dc9ea80-315c-4d96-b62b-5089226a9f65/registry/0.log"
Oct 30 15:11:57.886749 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:57.886717    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6pbvr_08311d3a-d790-4761-96f0-e9a6b336e913/node-ca/0.log"
Oct 30 15:11:57.922041 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:57.922006    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hdsbg_2e761f65-785b-45e0-90df-5ea5697d281b/discovery/0.log"
Oct 30 15:11:58.022808 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:58.022778    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8996685d-q52jz_cba183dd-ad5b-469f-9058-67196f9557e5/kube-auth-proxy/0.log"
Oct 30 15:11:58.093706 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:58.093680    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-52g42_075cda4c-17d3-48d0-801c-0a0101cb3a5b/istio-proxy/0.log"
Oct 30 15:11:58.298276 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:58.298244    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-d9pw9_b9c516e0-9b3d-4398-a08b-c9db1a762db1/istio-proxy/0.log"
Oct 30 15:11:58.432846 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:58.432756    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-fb779_588b499d-571d-449a-9e75-126e4c84902c/istio-proxy/0.log"
Oct 30 15:11:58.699441 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:58.699360    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-p5gtn_7d2c5de7-2259-4764-bca8-8811fe92c03d/istio-proxy/0.log"
Oct 30 15:11:58.768753 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:58.768720    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-g99dv_08de2a9c-9802-4b72-a844-58f650b6e573/payload-pre-processing/0.log"
Oct 30 15:11:58.836244 ip-10-0-133-118 kubenswrapper[4530]: I1030 15:11:58.836207    4530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-9dw4d_c340cff3-c8aa-40df-869a-5d5bf4c7210b/payload-processing/0.log"